[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:35 dist-test-slave-21p4 krb5kdc[12259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:35 dist-test-slave-21p4 krb5kdc[12259](info): set up 2 sockets
Sep 26 05:46:35 dist-test-slave-21p4 krb5kdc[12259](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 05:46:36.310956 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:36.434387 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:41227
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:36.518814 12272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:36.519032 12272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:36.519057 12272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:36.520262 12272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:36.520304 12272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:36.520318 12272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:36.520331 12272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:36.520344 12272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:36.521811 12272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:36.522018 12272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:36.522192 12272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:36.527287 12278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:36.527361 12280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:36.527482 12272 server_base.cc:1034] running on GCE node
W20260926 05:46:36.527565 12277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:36.527933 12272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:36.528287 12272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:36.529500 12272 hybrid_clock.cc:648] HybridClock initialized: now 1790401596529488 us; error 52 us; skew 500 ppm
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:36.532277 12272 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:36.532805 12272 webserver.cc:460] Webserver started at http://127.11.247.62:32889/ using document root <none> and password file <none>
I20260926 05:46:36.533066 12272 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:36.533139 12272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:36.533299 12272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:36.534449 12272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "5277c0e7257a4b94bb8c9345c1f7f964"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "832ebb43a9837f633e92282906990c4a"
server_key_iv: "daa851993dcf80fdd3f0021aec15db37"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.534909 12272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "5277c0e7257a4b94bb8c9345c1f7f964"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "832ebb43a9837f633e92282906990c4a"
server_key_iv: "daa851993dcf80fdd3f0021aec15db37"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.536361 12272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:36.537189 12287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.537380 12272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:36.537487 12272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "5277c0e7257a4b94bb8c9345c1f7f964"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "832ebb43a9837f633e92282906990c4a"
server_key_iv: "daa851993dcf80fdd3f0021aec15db37"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.537586 12272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:36.545320 12272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:36.547271 12272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:36.547431 12272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:36.551760 12272 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:41227
I20260926 05:46:36.551815 12339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:41227 every 8 connection(s)
I20260926 05:46:36.552196 12272 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:36.553098 12340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:36.555475 12340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964: Bootstrap starting.
I20260926 05:46:36.556072 12340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:36.556360 12340 log.cc:824] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:36.557163 12340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964: No bootstrap required, opened a new log
I20260926 05:46:36.558876 12340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41227 } }
I20260926 05:46:36.559023 12340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:36.559046 12340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5277c0e7257a4b94bb8c9345c1f7f964, State: Initialized, Role: FOLLOWER
I20260926 05:46:36.559211 12340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41227 } }
I20260926 05:46:36.559290 12340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:36.559330 12340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:36.559402 12340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:36.559607 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 12272
I20260926 05:46:36.559687 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:36.560037 12340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41227 } }
I20260926 05:46:36.560156 12340 leader_election.cc:302] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5277c0e7257a4b94bb8c9345c1f7f964; no voters:
I20260926 05:46:36.560361 12340 leader_election.cc:288] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:36.560380 12343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:36.560397 12252 external_mini_cluster.cc:1428] Setting key a904916983a9554914b802032cb32660
I20260926 05:46:36.560497 12343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [term 1 LEADER]: Becoming Leader. State: Replica: 5277c0e7257a4b94bb8c9345c1f7f964, State: Running, Role: LEADER
I20260926 05:46:36.560582 12343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41227 } }
I20260926 05:46:36.560632 12340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:36.560894 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41227 } } }
I20260926 05:46:36.561041 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:36.561292 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5277c0e7257a4b94bb8c9345c1f7f964. Latest consensus state: current_term: 1 leader_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5277c0e7257a4b94bb8c9345c1f7f964" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41227 } } }
I20260926 05:46:36.561414 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:36.561892 12350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:36.562515 12350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:36.564385 12350 catalog_manager.cc:1347] Generated new cluster ID: 0386e62b538a49c9adfff03499298a62
I20260926 05:46:36.564443 12350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:36.574120 12350 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:36.574807 12350 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:36.578774 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:36.561857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.11.247.62:41227)
0926 05:46:36.562478 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:36.562486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:36.562513 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:46:36.564275 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:36.564283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:36.564344 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:36.564495 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:36.566432 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.567958 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.569435 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.570086 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.570208 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:36.574561 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:36.574567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:36.574567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:36.574607 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:36.576730 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:36.577228 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:36.577232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:36.577232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:36.577350 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:36.577619 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:36.577620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:36.577620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:36.577908 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:36.578055 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:36.578359 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 05:46:36.578510 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":64,"threads_started":1}
I20260926 05:46:36.581059 12350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5277c0e7257a4b94bb8c9345c1f7f964: Generated new TSK 0
I20260926 05:46:36.581274 12350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:36.610719 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41227
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:36.696022 12366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:36.696233 12366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:36.696261 12366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:36.697438 12366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:36.697484 12366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:36.697513 12366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:36.699090 12366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:36.699348 12366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:36.699645 12366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:36.704943 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:36.704975 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:36.704962 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:36.705021 12366 server_base.cc:1034] running on GCE node
I20260926 05:46:36.705448 12366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:36.705706 12366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:36.706854 12366 hybrid_clock.cc:648] HybridClock initialized: now 1790401596706832 us; error 33 us; skew 500 ppm
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:36.709319 12366 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:36.709756 12366 webserver.cc:460] Webserver started at http://127.11.247.1:37909/ using document root <none> and password file <none>
I20260926 05:46:36.709986 12366 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:36.710054 12366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:36.710210 12366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:36.711409 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "26e885b157714dddaef5ad50a7fc46fb"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "0efa3b6c0c11080734672b59b3bd5ae6"
server_key_iv: "c0c243176d18a17817cfea3a97756cd3"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.711796 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "26e885b157714dddaef5ad50a7fc46fb"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "0efa3b6c0c11080734672b59b3bd5ae6"
server_key_iv: "c0c243176d18a17817cfea3a97756cd3"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.713145 12366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:46:36.713901 12381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.714068 12366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:36.714172 12366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "26e885b157714dddaef5ad50a7fc46fb"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "0efa3b6c0c11080734672b59b3bd5ae6"
server_key_iv: "c0c243176d18a17817cfea3a97756cd3"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.714248 12366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:36.723490 12366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:36.725409 12366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:36.725582 12366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:36.725821 12366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:36.726141 12366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:36.726199 12366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.726240 12366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:36.726279 12366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.732419 12366 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:39663
I20260926 05:46:36.732460 12494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:39663 every 8 connection(s)
I20260926 05:46:36.732806 12366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:36.735749 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 12366
I20260926 05:46:36.735854 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:36.736029 12252 external_mini_cluster.cc:1428] Setting key 24d01146263b222d1e4d0173999770cc
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:36.744093 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:36.733623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:54285 (local address 127.11.247.62:41227)
0926 05:46:36.733762 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:36.733765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:36.733976 (+ 211us) server_negotiation.cc:407] Connection header received
0926 05:46:36.734542 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:36.734543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:36.734557 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:36.734654 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:36.735842 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.736517 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.738047 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.738247 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.738341 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:36.741000 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:36.741001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:36.741001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:36.741008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:36.742502 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:36.743055 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:36.743056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:36.743057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:36.743112 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:36.743405 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:36.743405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:36.743406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:36.743513 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:36.743568 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:36.743931 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 05:46:36.744016 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 05:46:36.744379 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:36.733605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41227 (local address 127.11.247.1:54285)
0926 05:46:36.733817 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:36.733876 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:36.734423 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:36.734736 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:36.734741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:36.735397 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:36.735708 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:36.735716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.736603 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.736604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:36.737958 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:36.737961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.738353 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.738355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:36.738499 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:36.739109 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:36.739117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:36.740904 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:36.742608 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:36.742617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:36.742619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:36.742975 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:36.743180 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:36.743181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:36.743181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:36.743335 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:36.743585 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:36.743592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:36.743781 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:46:36.743911 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 05:46:36.744101 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260926 05:46:36.744962 12495 heartbeater.cc:347] Connected to a master server at 127.11.247.62:41227
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
I20260926 05:46:36.745081 12495 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:36.745293 12495 heartbeater.cc:511] Master 127.11.247.62:41227 requested a full tablet report, sending...
I20260926 05:46:36.745857 12304 ts_manager.cc:194] Registered new tserver with Master: 26e885b157714dddaef5ad50a7fc46fb (127.11.247.1:39663)
I20260926 05:46:36.746505 12304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:54285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:36.765130 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41227
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:36.848086 12500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:36.848336 12500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:36.848366 12500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:36.849658 12500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:36.849747 12500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:36.849800 12500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:36.851428 12500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:36.851724 12500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:36.851992 12500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:46:36.857514 12500 server_base.cc:1034] running on GCE node
W20260926 05:46:36.857481 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:36.857481 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:36.857479 12505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:36.857980 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:36.858269 12500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:36.859448 12500 hybrid_clock.cc:648] HybridClock initialized: now 1790401596859431 us; error 28 us; skew 500 ppm
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:36.862396 12500 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:36.862959 12500 webserver.cc:460] Webserver started at http://127.11.247.2:39983/ using document root <none> and password file <none>
I20260926 05:46:36.863165 12500 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:36.863209 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:36.863315 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:36.864436 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "5262d6ed9e154009b4ec916a55bddca2"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "d60814a14246b4c161e87b7ca0278a17"
server_key_iv: "8c1de0189aa1e1de70ec41d28bc07765"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.864760 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "5262d6ed9e154009b4ec916a55bddca2"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "d60814a14246b4c161e87b7ca0278a17"
server_key_iv: "8c1de0189aa1e1de70ec41d28bc07765"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.866182 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:36.867090 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.867278 12500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:46:36.867383 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "5262d6ed9e154009b4ec916a55bddca2"
format_stamp: "Formatted at 2026-09-26 05:46:36 on dist-test-slave-21p4"
server_key: "d60814a14246b4c161e87b7ca0278a17"
server_key_iv: "8c1de0189aa1e1de70ec41d28bc07765"
server_key_version: "encryptionkey@0"
I20260926 05:46:36.867447 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:36.882042 12500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:36.884199 12500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:36.884325 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:36.884529 12500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:36.884887 12500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:36.884922 12500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.884948 12500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:36.884960 12500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:36.890926 12500 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:37923
I20260926 05:46:36.891007 12628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:37923 every 8 connection(s)
I20260926 05:46:36.891396 12500 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:46:36 dist-test-slave-21p4 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401596, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:36.900764 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 12500
I20260926 05:46:36.900873 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:36.901088 12252 external_mini_cluster.cc:1428] Setting key fc223e8b686c9eeb4bc251568a0da03d
I20260926 05:46:36.901778 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:36.892293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:37217 (local address 127.11.247.62:41227)
0926 05:46:36.892357 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:36.892360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:36.892556 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:46:36.893139 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:36.893141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:36.893154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:36.893214 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:36.894259 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.894876 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.895982 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.896191 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.896271 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:36.898647 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:36.898648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:36.898648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:36.898656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:36.900050 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:36.900550 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:36.900551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:36.900551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:36.900613 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:36.900999 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:36.901000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:36.901001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:36.901164 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:36.901242 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:36.901589 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 05:46:36.901677 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:46:36.902091 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:36.892257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41227 (local address 127.11.247.2:37217)
0926 05:46:36.892424 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:36.892471 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:36.893019 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:36.893297 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:36.893302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:36.893912 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:36.894150 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:36.894155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.894974 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.894975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:36.895864 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:36.895867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:36.896284 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:36.896285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:36.896417 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:36.896961 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:36.896969 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:36.898494 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:36.900174 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:36.900182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:36.900184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:36.900439 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:36.900690 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:36.900691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:36.900691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:36.900861 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:36.901232 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:36.901239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:36.901429 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 05:46:36.901584 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 05:46:36.901812 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 05:46:36.902648 12629 heartbeater.cc:347] Connected to a master server at 127.11.247.62:41227
I20260926 05:46:36.902750 12629 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:36.902966 12629 heartbeater.cc:511] Master 127.11.247.62:41227 requested a full tablet report, sending...
I20260926 05:46:36.903414 12304 ts_manager.cc:194] Registered new tserver with Master: 5262d6ed9e154009b4ec916a55bddca2 (127.11.247.2:37923)
I20260926 05:46:36.903832 12304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:37217
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:36.931003 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41227
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:37.016104 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:37.016368 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:37.016415 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:37.017694 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:37.017745 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:37.017773 12634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:37.019258 12634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:37.019472 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:37.019662 12634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:37.024663 12639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:37.024681 12640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:37.024745 12642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:37.024926 12634 server_base.cc:1034] running on GCE node
I20260926 05:46:37.025143 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:37.025416 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:37.026629 12634 hybrid_clock.cc:648] HybridClock initialized: now 1790401597026620 us; error 37 us; skew 500 ppm
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401597, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:37.029305 12634 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:37.029814 12634 webserver.cc:460] Webserver started at http://127.11.247.3:44977/ using document root <none> and password file <none>
I20260926 05:46:37.030076 12634 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:37.030151 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:37.030303 12634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:37.031523 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "0d11726253b644bf9bf07a9a9ead9977"
format_stamp: "Formatted at 2026-09-26 05:46:37 on dist-test-slave-21p4"
server_key: "3514c4be8d0165c0e273f63c12f5bf65"
server_key_iv: "e0162e19f4bc347ba3f5929dff800908"
server_key_version: "encryptionkey@0"
I20260926 05:46:37.031930 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "0d11726253b644bf9bf07a9a9ead9977"
format_stamp: "Formatted at 2026-09-26 05:46:37 on dist-test-slave-21p4"
server_key: "3514c4be8d0165c0e273f63c12f5bf65"
server_key_iv: "e0162e19f4bc347ba3f5929dff800908"
server_key_version: "encryptionkey@0"
I20260926 05:46:37.033340 12634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:46:37.034138 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:37.034334 12634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:37.034442 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "0d11726253b644bf9bf07a9a9ead9977"
format_stamp: "Formatted at 2026-09-26 05:46:37 on dist-test-slave-21p4"
server_key: "3514c4be8d0165c0e273f63c12f5bf65"
server_key_iv: "e0162e19f4bc347ba3f5929dff800908"
server_key_version: "encryptionkey@0"
I20260926 05:46:37.034521 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:37.064654 12634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:37.066761 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:37.066926 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:37.067168 12634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:37.067540 12634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:37.067574 12634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:37.067617 12634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:37.067643 12634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:37.073699 12634 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:37601
I20260926 05:46:37.073763 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:37601 every 8 connection(s)
I20260926 05:46:37.074101 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:37.076659 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 12634
I20260926 05:46:37.076784 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:37.076958 12252 external_mini_cluster.cc:1428] Setting key 1f3eee94a72b4feac859dc1638df954f
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401597, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:37.084365 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:37.074917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:34093 (local address 127.11.247.62:41227)
0926 05:46:37.074980 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:37.074983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:37.075198 (+ 215us) server_negotiation.cc:407] Connection header received
0926 05:46:37.075686 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:37.075687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:37.075697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:37.075759 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:37.076732 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.077526 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.078867 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.079163 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.079237 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:37.081494 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:37.081495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:37.081496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:37.081504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:37.083074 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:37.083476 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:37.083478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:37.083478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:37.083531 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:37.083765 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:37.083766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:37.083766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:37.083893 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:37.083943 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:37.084210 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 05:46:37.084294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:46:37.084560 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:37.074903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41227 (local address 127.11.247.3:34093)
0926 05:46:37.075066 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:37.075109 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:37.075598 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:37.075813 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:37.075817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:37.076404 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:37.076635 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:37.076640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.077614 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.077615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:37.078791 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:37.078794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.079250 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.079251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:37.079353 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:37.079841 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:37.079854 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:37.081368 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:37.083164 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:37.083171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:37.083173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:37.083398 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:37.083588 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:37.083589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:37.083589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:37.083708 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:37.083950 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:37.083954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:37.084082 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 05:46:37.084185 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 05:46:37.084314 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260926 05:46:37.085080 12763 heartbeater.cc:347] Connected to a master server at 127.11.247.62:41227
I20260926 05:46:37.085194 12763 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:37.085383 12763 heartbeater.cc:511] Master 127.11.247.62:41227 requested a full tablet report, sending...
I20260926 05:46:37.085791 12304 ts_manager.cc:194] Registered new tserver with Master: 0d11726253b644bf9bf07a9a9ead9977 (127.11.247.3:37601)
I20260926 05:46:37.086170 12304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:34093
I20260926 05:46:37.088917 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:37.113037 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:37.104614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.11.247.1:39663)
0926 05:46:37.104784 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:37.104790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:37.104805 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:46:37.104847 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:37.104850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:37.104865 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:37.104934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:37.105534 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.106286 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.107110 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.107466 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.107525 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:37.109791 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:37.109802 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:37.109802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:37.109809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:37.111584 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:37.111958 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:37.111960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:37.111961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:37.112056 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:37.112417 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:37.112418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:37.112418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:37.112572 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:37.112681 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:37.112819 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:46:37.112890 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
W20260926 05:46:37.113420 12409 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36798
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:37.136790 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:37.129194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.11.247.1:39663)
0926 05:46:37.129281 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:37.129283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:37.129370 (+ 87us) server_negotiation.cc:407] Connection header received
0926 05:46:37.129486 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:37.129487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:37.129501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:37.129565 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:37.130270 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.130874 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.131694 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:37.131916 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:37.131974 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:37.134323 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:37.134324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:37.134325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:37.134333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:37.135628 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:37.135983 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:37.135985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:37.135985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:37.136041 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:37.136240 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:37.136240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:37.136241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:37.136346 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:37.136464 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:37.136582 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 05:46:37.136669 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:46:37.137461 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 12366
I20260926 05:46:37.142683 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 12500
I20260926 05:46:37.149166 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 12634
I20260926 05:46:37.154000 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 12272
2026-09-26T05:46:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1912 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12777](info): set up 2 sockets
Sep 26 05:46:37 dist-test-slave-21p4 krb5kdc[12777](info): commencing operation
krb5kdc: starting...
W20260926 05:46:38.190272 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:38.313915 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37951
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:38.396641 12790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:38.396889 12790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:38.396926 12790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:38.398108 12790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:38.398180 12790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:38.398203 12790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:38.398226 12790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:38.398253 12790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:38.399780 12790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:38.400060 12790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:38.400297 12790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:38.405351 12796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.405450 12790 server_base.cc:1034] running on GCE node
W20260926 05:46:38.405356 12795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:38.405356 12798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.405846 12790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:38.406152 12790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:38.407341 12790 hybrid_clock.cc:648] HybridClock initialized: now 1790401598407322 us; error 47 us; skew 500 ppm
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:38.409896 12790 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:38.410354 12790 webserver.cc:460] Webserver started at http://127.11.247.62:39261/ using document root <none> and password file <none>
I20260926 05:46:38.410669 12790 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:38.410743 12790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:38.410880 12790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:38.411952 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "1124892e8ef04d548950562172706b6f"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "f146890d9d3226e9c98d163c794544e3"
server_key_iv: "a66cbd0d675cc5b0e2cf3c531f014070"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.412328 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "1124892e8ef04d548950562172706b6f"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "f146890d9d3226e9c98d163c794544e3"
server_key_iv: "a66cbd0d675cc5b0e2cf3c531f014070"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.413700 12790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:38.414542 12805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.414775 12790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:38.414875 12790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "1124892e8ef04d548950562172706b6f"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "f146890d9d3226e9c98d163c794544e3"
server_key_iv: "a66cbd0d675cc5b0e2cf3c531f014070"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.414959 12790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:38.437354 12790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:38.439711 12790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:38.440095 12790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:38.444748 12790 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:37951
I20260926 05:46:38.444779 12857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:37951 every 8 connection(s)
I20260926 05:46:38.445145 12790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:38.446029 12858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:38.448287 12858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f: Bootstrap starting.
I20260926 05:46:38.448829 12858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:38.449162 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 12790
I20260926 05:46:38.449205 12858 log.cc:824] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:38.449249 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:38.449430 12252 external_mini_cluster.cc:1428] Setting key db6ca327b7180cc3e3a73c16536f6ec9
I20260926 05:46:38.450050 12858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f: No bootstrap required, opened a new log
I20260926 05:46:38.452873 12858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124892e8ef04d548950562172706b6f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37951 } }
I20260926 05:46:38.453084 12858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:38.453125 12858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1124892e8ef04d548950562172706b6f, State: Initialized, Role: FOLLOWER
I20260926 05:46:38.453254 12858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124892e8ef04d548950562172706b6f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37951 } }
I20260926 05:46:38.453330 12858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:38.453359 12858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:38.453397 12858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:38.454128 12858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124892e8ef04d548950562172706b6f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37951 } }
I20260926 05:46:38.454257 12858 leader_election.cc:302] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1124892e8ef04d548950562172706b6f; no voters:
I20260926 05:46:38.454463 12858 leader_election.cc:288] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:38.454492 12863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:38.454667 12863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [term 1 LEADER]: Becoming Leader. State: Replica: 1124892e8ef04d548950562172706b6f, State: Running, Role: LEADER
I20260926 05:46:38.454764 12863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124892e8ef04d548950562172706b6f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37951 } }
I20260926 05:46:38.455128 12864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1124892e8ef04d548950562172706b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124892e8ef04d548950562172706b6f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37951 } } }
I20260926 05:46:38.455155 12865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1124892e8ef04d548950562172706b6f. Latest consensus state: current_term: 1 leader_uuid: "1124892e8ef04d548950562172706b6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1124892e8ef04d548950562172706b6f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37951 } } }
I20260926 05:46:38.455238 12864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:38.455265 12865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:38.455127 12858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:38.455513 12867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:38.456069 12867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:38.457764 12867 catalog_manager.cc:1347] Generated new cluster ID: b37321aa773644b99f8686f51e47eed2
I20260926 05:46:38.457819 12867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:46:38.461129 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.449997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.11.247.62:37951)
0926 05:46:38.450154 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.450164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.450198 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:46:38.450680 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.450687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.450739 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.450854 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:38.451698 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.452647 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.453741 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.454179 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.454334 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.457360 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:38.457365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:38.457365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:38.457380 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:38.459471 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.459866 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.459869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.459870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.459983 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.460187 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.460188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.460189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.460513 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:38.460679 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.460759 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.460902 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":100,"threads_started":1}
I20260926 05:46:38.474071 12867 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:38.474766 12867 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:38.485004 12867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1124892e8ef04d548950562172706b6f: Generated new TSK 0
I20260926 05:46:38.485271 12867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:38.511624 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37951
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:38.595674 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:38.595896 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:38.595950 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:38.597235 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:38.597285 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:38.597312 12884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:38.598834 12884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37951
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:38.599067 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:38.599258 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:38.604372 12892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.604460 12884 server_base.cc:1034] running on GCE node
W20260926 05:46:38.604372 12890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:38.604380 12889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.604810 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:38.605113 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:38.606303 12884 hybrid_clock.cc:648] HybridClock initialized: now 1790401598606293 us; error 34 us; skew 500 ppm
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:38.609138 12884 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:38.609673 12884 webserver.cc:460] Webserver started at http://127.11.247.1:36955/ using document root <none> and password file <none>
I20260926 05:46:38.609920 12884 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:38.609990 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:38.610153 12884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:38.611428 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "a751bedebb93490180032f78014ebabc"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "8e3d62478ebfd33a9c7ad2add547bf06"
server_key_iv: "0cf0199d2f37570771469874abd3068e"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.611825 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "a751bedebb93490180032f78014ebabc"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "8e3d62478ebfd33a9c7ad2add547bf06"
server_key_iv: "0cf0199d2f37570771469874abd3068e"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.613277 12884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:38.614220 12899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.614421 12884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:38.614667 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "a751bedebb93490180032f78014ebabc"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "8e3d62478ebfd33a9c7ad2add547bf06"
server_key_iv: "0cf0199d2f37570771469874abd3068e"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.614758 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:38.636561 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:38.638770 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:38.638938 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:38.639197 12884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:38.639518 12884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:38.639568 12884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.639609 12884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:38.639647 12884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.645361 12884 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:38557
I20260926 05:46:38.645408 13012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:38557 every 8 connection(s)
I20260926 05:46:38.645732 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:38.646791 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 12884
I20260926 05:46:38.646903 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:38.647086 12252 external_mini_cluster.cc:1428] Setting key a417486da495f910b650f887ff6d952c
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
I20260926 05:46:38.657207 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.646480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:58591 (local address 127.11.247.62:37951)
0926 05:46:38.646544 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.646546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.646829 (+ 283us) server_negotiation.cc:407] Connection header received
0926 05:46:38.647991 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.647992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.648004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.648049 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:38.648682 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.649498 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.651057 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.651303 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.651374 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.653919 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:38.653921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:38.653921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:38.653929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:38.655634 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.656104 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.656105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.656106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.656167 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.656457 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.656458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.656458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.656578 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:38.656630 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.656986 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.657109 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:46:38.657435 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.646440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37951 (local address 127.11.247.1:58591)
0926 05:46:38.646693 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:38.646746 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:38.647893 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:38.648120 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:38.648124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:38.648389 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:38.648580 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.648584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.649646 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.649647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.650969 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.650972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.651388 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.651389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.651583 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.652129 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:38.652140 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:38.653822 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:38.655721 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:38.655728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:38.655730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:38.656020 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:38.656242 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:38.656242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:38.656243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:38.656406 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:38.656660 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:38.656667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:38.656813 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:46:38.656959 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 05:46:38.657150 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":179,"threads_started":1}
I20260926 05:46:38.658034 13013 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37951
I20260926 05:46:38.658150 13013 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:38.658353 13013 heartbeater.cc:511] Master 127.11.247.62:37951 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
I20260926 05:46:38.658938 12822 ts_manager.cc:194] Registered new tserver with Master: a751bedebb93490180032f78014ebabc (127.11.247.1:38557)
I20260926 05:46:38.659889 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:58591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:38.673709 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37951
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:38.756028 13018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:38.756253 13018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:38.756278 13018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:38.757603 13018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:38.757655 13018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:38.757687 13018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:38.759393 13018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37951
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:38.759672 13018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:38.759920 13018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:38.764955 13023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:38.764958 13026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:38.764957 13024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.765285 13018 server_base.cc:1034] running on GCE node
I20260926 05:46:38.765559 13018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:38.765822 13018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:38.767011 13018 hybrid_clock.cc:648] HybridClock initialized: now 1790401598766997 us; error 54 us; skew 500 ppm
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:38.769737 13018 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:38.770187 13018 webserver.cc:460] Webserver started at http://127.11.247.2:35975/ using document root <none> and password file <none>
I20260926 05:46:38.770431 13018 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:38.770499 13018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:38.770676 13018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:38.771759 13018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "691d7012cb7943b0af366f6b2b150623"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "6b21b21f0ee33c0146aeb2e39eb10e6f"
server_key_iv: "9e9f1ee2f444ac83d9ba9e9efa2baf85"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.772113 13018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "691d7012cb7943b0af366f6b2b150623"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "6b21b21f0ee33c0146aeb2e39eb10e6f"
server_key_iv: "9e9f1ee2f444ac83d9ba9e9efa2baf85"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.773465 13018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:38.774294 13033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.774516 13018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:38.774680 13018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "691d7012cb7943b0af366f6b2b150623"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "6b21b21f0ee33c0146aeb2e39eb10e6f"
server_key_iv: "9e9f1ee2f444ac83d9ba9e9efa2baf85"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.774762 13018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:38.786516 13018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:38.788971 13018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:38.789151 13018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:38.789402 13018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:38.789733 13018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:38.789785 13018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.789826 13018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:38.789853 13018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.795629 13018 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:42233
I20260926 05:46:38.795759 13146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:42233 every 8 connection(s)
I20260926 05:46:38.796037 13018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:38.798800 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13018
I20260926 05:46:38.798897 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:38.799053 12252 external_mini_cluster.cc:1428] Setting key 410b983524c9162b6c8498c9b49b2445
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:38.806753 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.796784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:42043 (local address 127.11.247.62:37951)
0926 05:46:38.796828 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.796830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.797043 (+ 213us) server_negotiation.cc:407] Connection header received
0926 05:46:38.798028 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.798030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.798044 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.798108 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:38.798841 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.799534 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.800592 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.800797 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.800868 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.803434 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:38.803436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:38.803436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:38.803445 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:38.805195 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.805658 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.805660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.805660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.805729 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.805988 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.805989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.805989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.806154 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:38.806227 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.806484 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.806628 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:46:38.806886 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.796752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37951 (local address 127.11.247.2:42043)
0926 05:46:38.796906 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:38.796944 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:38.797901 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:38.798187 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:38.798191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:38.798464 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:38.798753 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.798759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.799630 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.799631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.800476 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.800479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.800887 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.800888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.801031 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.801634 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:38.801644 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:38.803317 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:38.805298 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:38.805306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:38.805308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:38.805551 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:38.805807 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:38.805808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:38.805808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:38.805923 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:38.806220 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:38.806224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:38.806353 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 05:46:38.806440 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:46:38.806639 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260926 05:46:38.807449 13147 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37951
I20260926 05:46:38.807545 13147 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:38.807773 13147 heartbeater.cc:511] Master 127.11.247.62:37951 requested a full tablet report, sending...
I20260926 05:46:38.808228 12822 ts_manager.cc:194] Registered new tserver with Master: 691d7012cb7943b0af366f6b2b150623 (127.11.247.2:42233)
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:46:38.808746 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:42043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:38.826671 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37951
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:38.909130 13152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:38.909353 13152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:38.909379 13152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:38.910509 13152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:38.910547 13152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:38.910625 13152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:38.912194 13152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37951
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:38.912405 13152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:38.912595 13152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:38.917970 13158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:38.917979 13160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.917995 13152 server_base.cc:1034] running on GCE node
W20260926 05:46:38.917989 13157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:38.918468 13152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:38.918741 13152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:38.919884 13152 hybrid_clock.cc:648] HybridClock initialized: now 1790401598919872 us; error 29 us; skew 500 ppm
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:38.922346 13152 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:38.922860 13152 webserver.cc:460] Webserver started at http://127.11.247.3:39133/ using document root <none> and password file <none>
I20260926 05:46:38.923108 13152 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:38.923175 13152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:38.923317 13152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:38.924391 13152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "34d0d6cf9b5042fdbc151e03fa58856f"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "f50c1636d500ab8d27d0511e448be10c"
server_key_iv: "63ece0f8ecc227d537f172e70cfebaa8"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.924759 13152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "34d0d6cf9b5042fdbc151e03fa58856f"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "f50c1636d500ab8d27d0511e448be10c"
server_key_iv: "63ece0f8ecc227d537f172e70cfebaa8"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.926108 13152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:46:38.926892 13167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.927052 13152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:38.927141 13152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "34d0d6cf9b5042fdbc151e03fa58856f"
format_stamp: "Formatted at 2026-09-26 05:46:38 on dist-test-slave-21p4"
server_key: "f50c1636d500ab8d27d0511e448be10c"
server_key_iv: "63ece0f8ecc227d537f172e70cfebaa8"
server_key_version: "encryptionkey@0"
I20260926 05:46:38.927217 13152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:38.939842 13152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:38.942209 13152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:38.942389 13152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:38.942786 13152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:38.943149 13152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:38.943187 13152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.943243 13152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:38.943275 13152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:38.949564 13152 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:42085
I20260926 05:46:38.949630 13280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:42085 every 8 connection(s)
I20260926 05:46:38.950021 13152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:38.951745 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13152
I20260926 05:46:38.951884 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:38.952124 12252 external_mini_cluster.cc:1428] Setting key df263c1cff2a81a70dfa7b346ea1cb26
Sep 26 05:46:38 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:38.960480 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.950843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45137 (local address 127.11.247.62:37951)
0926 05:46:38.950893 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.950896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.951146 (+ 250us) server_negotiation.cc:407] Connection header received
0926 05:46:38.952189 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.952191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.952214 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.952304 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:38.953219 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.953751 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.954837 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.955025 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.955081 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.957565 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:38.957567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:38.957567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:38.957581 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:38.958954 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.959453 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.959454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.959455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.959515 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.959793 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.959794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.959794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.959923 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:38.959991 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.960296 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.960383 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:46:38.960732 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.950811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37951 (local address 127.11.247.3:45137)
0926 05:46:38.950990 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:38.951035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:38.952076 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:38.952395 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:38.952400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:38.952819 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:38.953122 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.953129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.953830 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.953830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.954755 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.954757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.955096 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.955097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.955245 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.955821 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:38.955830 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:38.957441 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:38.959065 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:38.959073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:38.959076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:38.959351 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:38.959580 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:38.959581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:38.959581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:38.959717 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:38.959991 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:38.959999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:38.960149 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 05:46:38.960260 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:46:38.960437 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260926 05:46:38.961207 13281 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37951
I20260926 05:46:38.961304 13281 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:38.961453 13281 heartbeater.cc:511] Master 127.11.247.62:37951 requested a full tablet report, sending...
I20260926 05:46:38.961848 12822 ts_manager.cc:194] Registered new tserver with Master: 34d0d6cf9b5042fdbc151e03fa58856f (127.11.247.3:42085)
I20260926 05:46:38.962248 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:45137
I20260926 05:46:38.963872 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:46:38.971769 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.965390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.11.247.62:37951)
0926 05:46:38.965444 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.965446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.965535 (+ 89us) server_negotiation.cc:407] Connection header received
0926 05:46:38.965618 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.965619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.965627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.965682 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:38.966216 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.966782 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.967477 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.967726 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.967790 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.969216 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:38.969217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:38.969217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:38.969224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:38.970517 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.970950 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.970951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.970952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.971013 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:38.971194 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:38.971195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:38.971195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:38.971368 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:38.971449 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.971616 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.971685 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:46:38.974248 12822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:38.976099 12822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:38.982677 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.978352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:42085 (local address 127.0.0.1:58202)
0926 05:46:38.978491 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:38.978535 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:38.978641 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:38.979018 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:38.979021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:38.979023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:38.979106 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.979108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.980586 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.980587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.981357 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.981359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.982297 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.982299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.982380 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.982394 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:46:38.982480 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:46:38.982544 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260926 05:46:38.982779 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.978775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.11.247.1:38557)
0926 05:46:38.978953 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.978958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.979090 (+ 132us) server_negotiation.cc:407] Connection header received
0926 05:46:38.979218 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.979221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.979237 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.979318 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:38.979458 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.980148 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.981653 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.982358 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.982422 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.982478 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.982644 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.982696 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260926 05:46:38.982832 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.978358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:42233 (local address 127.0.0.1:45958)
0926 05:46:38.978517 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:38.978535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:38.978684 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:38.979048 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:38.979049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:38.979051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:38.979116 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.979118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.980512 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.980513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.981628 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.981630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.982540 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.982541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.982646 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.982651 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:38.982712 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:46:38.982750 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260926 05:46:38.982944 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.978457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.11.247.3:42085)
0926 05:46:38.978801 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.978806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.978817 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:46:38.978860 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.978863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.978880 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.979001 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:38.979412 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.980459 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.981463 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.982209 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.982290 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.982353 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.982799 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.982851 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":83,"threads_started":1}
I20260926 05:46:38.983022 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.978704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.11.247.2:42233)
0926 05:46:38.978884 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:38.978887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:38.978897 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:38.978933 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:38.978935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:38.978949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:38.979189 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:38.979276 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.980405 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.981707 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.982439 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.982532 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.982618 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:38.982728 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 05:46:38.982919 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260926 05:46:38.983556 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:38.978568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:38557 (local address 127.0.0.1:35648)
0926 05:46:38.979006 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:38.979016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:38.979118 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:38.979316 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:38.979317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:38.979319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:38.979365 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.979366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.980273 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.980273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.981558 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:38.981561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:38.982437 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:38.982438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:38.982504 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:38.982509 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:38.983467 (+ 958us) client_negotiation.cc:240] Negotiation successful
0926 05:46:38.983508 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":32,"thread_start_us":368,"threads_started":1}
I20260926 05:46:38.983662 13081 tablet_service.cc:1467] Processing CreateTablet for tablet 1b90b33a376d4f5daad6937d73877375 (DEFAULT_TABLE table=test-table [id=650cf9249abe4bca8ac178b5fb5fd396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:38.983672 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 1b90b33a376d4f5daad6937d73877375 (DEFAULT_TABLE table=test-table [id=650cf9249abe4bca8ac178b5fb5fd396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:38.984035 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b90b33a376d4f5daad6937d73877375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:38.984035 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b90b33a376d4f5daad6937d73877375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:38.987506 13296 tablet_bootstrap.cc:492] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: Bootstrap starting.
I20260926 05:46:38.988265 13296 tablet_bootstrap.cc:654] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:38.988873 13296 log.cc:824] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:38.989238 13295 tablet_bootstrap.cc:492] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: Bootstrap starting.
I20260926 05:46:38.989212 12947 tablet_service.cc:1467] Processing CreateTablet for tablet 1b90b33a376d4f5daad6937d73877375 (DEFAULT_TABLE table=test-table [id=650cf9249abe4bca8ac178b5fb5fd396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:38.989634 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b90b33a376d4f5daad6937d73877375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:38.989658 13296 tablet_bootstrap.cc:492] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: No bootstrap required, opened a new log
I20260926 05:46:38.989737 13296 ts_tablet_manager.cc:1397] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:38.989910 13295 tablet_bootstrap.cc:654] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:38.990453 13295 log.cc:824] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:38.991285 13295 tablet_bootstrap.cc:492] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: No bootstrap required, opened a new log
I20260926 05:46:38.991356 13295 ts_tablet_manager.cc:1397] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:38.991597 13296 raft_consensus.cc:348] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:38.991760 13296 raft_consensus.cc:374] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:38.991803 13296 raft_consensus.cc:729] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d0d6cf9b5042fdbc151e03fa58856f, State: Initialized, Role: FOLLOWER
I20260926 05:46:38.991935 13296 consensus_queue.cc:260] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:38.992214 13296 ts_tablet_manager.cc:1428] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:46:38.992376 13281 heartbeater.cc:503] Master 127.11.247.62:37951 was elected leader, sending a full tablet report...
I20260926 05:46:38.993506 13300 tablet_bootstrap.cc:492] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: Bootstrap starting.
I20260926 05:46:38.993439 13295 raft_consensus.cc:348] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:38.993646 13295 raft_consensus.cc:374] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:38.993695 13295 raft_consensus.cc:729] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691d7012cb7943b0af366f6b2b150623, State: Initialized, Role: FOLLOWER
I20260926 05:46:38.993818 13295 consensus_queue.cc:260] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:38.994112 13295 ts_tablet_manager.cc:1428] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:46:38.994117 13147 heartbeater.cc:503] Master 127.11.247.62:37951 was elected leader, sending a full tablet report...
I20260926 05:46:38.994179 13300 tablet_bootstrap.cc:654] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:38.994791 13300 log.cc:824] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:38.995519 13300 tablet_bootstrap.cc:492] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: No bootstrap required, opened a new log
I20260926 05:46:38.995573 13300 ts_tablet_manager.cc:1397] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:46:38.996898 13300 raft_consensus.cc:348] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:38.997038 13300 raft_consensus.cc:374] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:38.997061 13300 raft_consensus.cc:729] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a751bedebb93490180032f78014ebabc, State: Initialized, Role: FOLLOWER
I20260926 05:46:38.997171 13300 consensus_queue.cc:260] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:38.997445 13300 ts_tablet_manager.cc:1428] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:46:38.997486 13013 heartbeater.cc:503] Master 127.11.247.62:37951 was elected leader, sending a full tablet report...
W20260926 05:46:39.046707 13148 tablet.cc:2369] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:46:39.122888 13303 raft_consensus.cc:482] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:39.123016 13303 raft_consensus.cc:504] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:39.123451 13303 leader_election.cc:288] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d0d6cf9b5042fdbc151e03fa58856f (127.11.247.3:42085), 691d7012cb7943b0af366f6b2b150623 (127.11.247.2:42233)
I20260926 05:46:39.126412 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.123669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:42233 (local address 127.11.247.1:59357)
0926 05:46:39.123722 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:39.123733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:39.123781 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:39.123947 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:39.123948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:39.123950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:39.124026 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:39.124028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.124745 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.124746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:39.125554 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:39.125556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.126166 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.126167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:39.126248 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.126253 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:39.126307 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 05:46:39.126357 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:46:39.126446 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.123736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:59357 (local address 127.11.247.2:42233)
0926 05:46:39.123789 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:39.123791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:39.123807 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:46:39.123841 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:39.123842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:39.123855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:39.123926 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:39.124093 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.124671 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.125666 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.126092 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.126151 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.126158 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:39.126322 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 05:46:39.126361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:46:39.126613 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.123774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:52227 (local address 127.11.247.3:42085)
0926 05:46:39.123823 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:39.123826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:39.124012 (+ 186us) server_negotiation.cc:407] Connection header received
0926 05:46:39.124046 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:39.124047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:39.124059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:39.124107 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:39.124243 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.124804 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.125803 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.126227 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.126303 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.126312 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:39.126480 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 05:46:39.126523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:46:39.126717 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.123680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:42085 (local address 127.11.247.1:52227)
0926 05:46:39.123935 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:39.123946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:39.123998 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:39.124120 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:39.124121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:39.124123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:39.124182 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:39.124183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.124897 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.124898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:39.125714 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:39.125716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.126328 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.126329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:39.126398 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.126402 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:39.126459 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 05:46:39.126497 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":94,"thread_start_us":98,"threads_started":1}
I20260926 05:46:39.126828 13101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90b33a376d4f5daad6937d73877375" candidate_uuid: "a751bedebb93490180032f78014ebabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691d7012cb7943b0af366f6b2b150623" is_pre_election: true
I20260926 05:46:39.127004 13101 raft_consensus.cc:2454] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a751bedebb93490180032f78014ebabc in term 0.
I20260926 05:46:39.127082 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90b33a376d4f5daad6937d73877375" candidate_uuid: "a751bedebb93490180032f78014ebabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" is_pre_election: true
I20260926 05:46:39.127231 13235 raft_consensus.cc:2454] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a751bedebb93490180032f78014ebabc in term 0.
I20260926 05:46:39.127233 12900 leader_election.cc:302] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 691d7012cb7943b0af366f6b2b150623, a751bedebb93490180032f78014ebabc; no voters:
I20260926 05:46:39.127352 13303 raft_consensus.cc:2781] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:39.127409 13303 raft_consensus.cc:482] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:39.127452 13303 raft_consensus.cc:3037] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:39.128275 13303 raft_consensus.cc:504] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:39.128418 13303 leader_election.cc:288] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [CANDIDATE]: Term 1 election: Requested vote from peers 34d0d6cf9b5042fdbc151e03fa58856f (127.11.247.3:42085), 691d7012cb7943b0af366f6b2b150623 (127.11.247.2:42233)
I20260926 05:46:39.128613 13101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90b33a376d4f5daad6937d73877375" candidate_uuid: "a751bedebb93490180032f78014ebabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691d7012cb7943b0af366f6b2b150623"
I20260926 05:46:39.128636 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b90b33a376d4f5daad6937d73877375" candidate_uuid: "a751bedebb93490180032f78014ebabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d0d6cf9b5042fdbc151e03fa58856f"
I20260926 05:46:39.128701 13101 raft_consensus.cc:3037] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:39.128729 13235 raft_consensus.cc:3037] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:39.129467 13235 raft_consensus.cc:2454] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a751bedebb93490180032f78014ebabc in term 1.
I20260926 05:46:39.129472 13101 raft_consensus.cc:2454] T 1b90b33a376d4f5daad6937d73877375 P 691d7012cb7943b0af366f6b2b150623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a751bedebb93490180032f78014ebabc in term 1.
I20260926 05:46:39.129629 12900 leader_election.cc:302] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 691d7012cb7943b0af366f6b2b150623, a751bedebb93490180032f78014ebabc; no voters:
I20260926 05:46:39.129719 13303 raft_consensus.cc:2781] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:39.129901 13303 raft_consensus.cc:686] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [term 1 LEADER]: Becoming Leader. State: Replica: a751bedebb93490180032f78014ebabc, State: Running, Role: LEADER
I20260926 05:46:39.129990 13303 consensus_queue.cc:237] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } }
I20260926 05:46:39.130899 12821 catalog_manager.cc:5495] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc reported cstate change: term changed from 0 to 1, leader changed from <none> to a751bedebb93490180032f78014ebabc (127.11.247.1). New cstate: current_term: 1 leader_uuid: "a751bedebb93490180032f78014ebabc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d0d6cf9b5042fdbc151e03fa58856f" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691d7012cb7943b0af366f6b2b150623" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a751bedebb93490180032f78014ebabc" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 38557 } health_report { overall_health: HEALTHY } } }
W20260926 05:46:39.146472 13014 tablet.cc:2369] T 1b90b33a376d4f5daad6937d73877375 P a751bedebb93490180032f78014ebabc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:39.170274 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.161429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.11.247.1:38557)
0926 05:46:39.161518 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:39.161521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:39.161657 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:46:39.161772 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:39.161773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:39.161789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:39.161882 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:39.162519 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.163298 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.164204 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.164468 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.164540 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.167037 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:39.167052 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:39.167053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:39.167061 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:39.168935 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:39.169307 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:39.169311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:39.169312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:39.169399 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:39.169643 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:39.169644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:39.169644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:39.169814 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:39.169942 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:39.170048 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 05:46:39.170133 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:39.193477 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.186208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.11.247.1:38557)
0926 05:46:39.186281 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:39.186283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:39.186350 (+ 67us) server_negotiation.cc:407] Connection header received
0926 05:46:39.186385 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:39.186386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:39.186398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:39.186458 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:39.187152 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.187653 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.188401 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.188589 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.188641 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.190856 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:39.190857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:39.190858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:39.190865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:39.192219 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:39.192614 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:39.192615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:39.192615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:39.192672 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:39.192898 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:39.192899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:39.192900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:39.193013 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:39.193101 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:39.193344 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 05:46:39.193410 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 05:46:39.200613 13282 tablet.cc:2369] T 1b90b33a376d4f5daad6937d73877375 P 34d0d6cf9b5042fdbc151e03fa58856f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:39.215878 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:39.209196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.11.247.1:38557)
0926 05:46:39.209263 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:39.209265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:39.209278 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:46:39.209349 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:39.209350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:39.209360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:39.209408 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:39.209964 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.210441 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.211223 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:39.211422 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:39.211477 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:39.213593 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:39.213594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:39.213595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:39.213602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:39.214890 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:39.215176 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:39.215176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:39.215177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:39.215228 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:39.215405 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:39.215406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:39.215406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:39.215513 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:39.215603 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:39.215709 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 05:46:39.215764 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 05:46:39.216380 12927 tablet_service.cc:2908] Rejecting scan request for tablet 1b90b33a376d4f5daad6937d73877375: Uninitialized: clean time has not yet been initialized
I20260926 05:46:39.217381 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 12884
I20260926 05:46:39.222364 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13018
I20260926 05:46:39.227243 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13152
I20260926 05:46:39.232184 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 12790
2026-09-26T05:46:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2079 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[13319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[13319](info): set up 2 sockets
Sep 26 05:46:39 dist-test-slave-21p4 krb5kdc[13319](info): commencing operation
krb5kdc: starting...
W20260926 05:46:40.354840 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 05:46:40.504913 12252 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.150s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 05:46:40.666390 12252 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.161s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/test-user.keytab.
W20260926 05:46:41.676118 12252 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.936s user 0.000s sys 0.001s
Sep 26 05:46:41 dist-test-slave-21p4 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
W20260926 05:46:42.653370 12252 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.247.62: real 0.954s user 0.000s sys 0.003s
I20260926 05:46:42.653712 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33335
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:42.737262 13332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:42.737495 13332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:42.737519 13332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:42.738783 13332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:42.738831 13332 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:46:42.738844 13332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:42.738857 13332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:42.738869 13332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:42.738881 13332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:42.740408 13332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:42.740643 13332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:42.740834 13332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:42.745872 13340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:42.745872 13338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:42.746002 13332 server_base.cc:1034] running on GCE node
W20260926 05:46:42.745877 13337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:42.746382 13332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:42.746712 13332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:42.747881 13332 hybrid_clock.cc:648] HybridClock initialized: now 1790401602747870 us; error 37 us; skew 500 ppm
Sep 26 05:46:42 dist-test-slave-21p4 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401602, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:42.756651 13332 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:42.757164 13332 webserver.cc:460] Webserver started at http://127.11.247.62:33231/ using document root <none> and password file <none>
I20260926 05:46:42.757431 13332 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:42.757500 13332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:42.757656 13332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:42.758895 13332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "67d6b4c21ea44225b65d7c9c976dc1e0"
format_stamp: "Formatted at 2026-09-26 05:46:42 on dist-test-slave-21p4"
server_key: "f086edec3f1712e634f92eb8c5a66650"
server_key_iv: "da1b4808655c1c2227011a6ebb3749d2"
server_key_version: "encryptionkey@0"
I20260926 05:46:42.759271 13332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "67d6b4c21ea44225b65d7c9c976dc1e0"
format_stamp: "Formatted at 2026-09-26 05:46:42 on dist-test-slave-21p4"
server_key: "f086edec3f1712e634f92eb8c5a66650"
server_key_iv: "da1b4808655c1c2227011a6ebb3749d2"
server_key_version: "encryptionkey@0"
I20260926 05:46:42.760725 13332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:42.761552 13347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:42.761862 13332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:42.761976 13332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "67d6b4c21ea44225b65d7c9c976dc1e0"
format_stamp: "Formatted at 2026-09-26 05:46:42 on dist-test-slave-21p4"
server_key: "f086edec3f1712e634f92eb8c5a66650"
server_key_iv: "da1b4808655c1c2227011a6ebb3749d2"
server_key_version: "encryptionkey@0"
I20260926 05:46:42.762058 13332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:42.775077 13332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:42.777508 13332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:42.777715 13332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:42.782855 13332 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33335
I20260926 05:46:42.782903 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33335 every 8 connection(s)
I20260926 05:46:42.783289 13332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:42.783974 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:42.786561 13410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: Bootstrap starting.
I20260926 05:46:42.787256 13410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:42.787554 13410 log.cc:824] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:42.788313 13410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: No bootstrap required, opened a new log
I20260926 05:46:42.788944 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13332
I20260926 05:46:42.789018 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:42.789193 12252 external_mini_cluster.cc:1428] Setting key daacc7c6153d38cc1ed30492ef8c4c7a
I20260926 05:46:42.790683 13410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33335 } }
I20260926 05:46:42.790874 13410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:42.790956 13410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d6b4c21ea44225b65d7c9c976dc1e0, State: Initialized, Role: FOLLOWER
I20260926 05:46:42.791138 13410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33335 } }
I20260926 05:46:42.791234 13410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:42.791275 13410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:42.791326 13410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:42.792088 13410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33335 } }
I20260926 05:46:42.792215 13410 leader_election.cc:302] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d6b4c21ea44225b65d7c9c976dc1e0; no voters:
I20260926 05:46:42.792392 13410 leader_election.cc:288] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:42.792452 13415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:42.792600 13415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [term 1 LEADER]: Becoming Leader. State: Replica: 67d6b4c21ea44225b65d7c9c976dc1e0, State: Running, Role: LEADER
I20260926 05:46:42.792654 13410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:42.792774 13415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33335 } }
I20260926 05:46:42.794140 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33335 } } }
I20260926 05:46:42.794246 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:42.794356 13416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d6b4c21ea44225b65d7c9c976dc1e0. Latest consensus state: current_term: 1 leader_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d6b4c21ea44225b65d7c9c976dc1e0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33335 } } }
I20260926 05:46:42.794441 13416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0 [sys.catalog]: This master's current role is: LEADER
W20260926 05:46:42.794852 13430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:42.795224 13430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:46:42.795305 13431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:42.795495 13431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:42.797173 13431 catalog_manager.cc:1347] Generated new cluster ID: 0397cc46efe74ac1a775d4f4f3c57178
I20260926 05:46:42.797225 13431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:46:42 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:42.802306 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:42.789770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.11.247.62:33335)
0926 05:46:42.789979 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:42.789990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:42.790032 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:46:42.790628 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:42.790637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:42.790707 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:42.790865 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:42.791647 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:42.793056 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:42.794212 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:42.794882 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:42.794968 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:42.797855 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:42.797860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:42.797861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:42.797873 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:42.800403 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:42.800911 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:42.800914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:42.800915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:42.801061 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:42.801299 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:42.801300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:42.801300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:42.801636 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:42.801794 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:42.801928 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 05:46:42.802074 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260926 05:46:42.816530 13431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:42.817202 13431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:42.824260 13431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: Generated new TSK 0
I20260926 05:46:42.824508 13431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:42.857865 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33335
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:42.942410 13436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:42.942685 13436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:42.942729 13436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:42.943955 13436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:42.944022 13436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:42.944060 13436 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:46:42.944111 13436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:42.945624 13436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:33335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:42.945946 13436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:42.946236 13436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:42.951452 13441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:42.951455 13444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:42.951457 13442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:42.951753 13436 server_base.cc:1034] running on GCE node
I20260926 05:46:42.952009 13436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:42.952268 13436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:42.953446 13436 hybrid_clock.cc:648] HybridClock initialized: now 1790401602953434 us; error 38 us; skew 500 ppm
Sep 26 05:46:42 dist-test-slave-21p4 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401602, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:42.956306 13436 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:42.956868 13436 webserver.cc:460] Webserver started at http://127.11.247.1:44385/ using document root <none> and password file <none>
I20260926 05:46:42.957123 13436 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:42.957192 13436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:42.957345 13436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:42.958864 13436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "82652eaeb41b4d419ee33f3127f76439"
format_stamp: "Formatted at 2026-09-26 05:46:42 on dist-test-slave-21p4"
server_key: "e1ddf74600cb739cb9ded538b0c2ad0a"
server_key_iv: "30b1ab0fa22d92733751b8242e3d20e4"
server_key_version: "encryptionkey@0"
I20260926 05:46:42.959314 13436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "82652eaeb41b4d419ee33f3127f76439"
format_stamp: "Formatted at 2026-09-26 05:46:42 on dist-test-slave-21p4"
server_key: "e1ddf74600cb739cb9ded538b0c2ad0a"
server_key_iv: "30b1ab0fa22d92733751b8242e3d20e4"
server_key_version: "encryptionkey@0"
I20260926 05:46:42.960877 13436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:42.961828 13451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:42.962030 13436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:42.962143 13436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "82652eaeb41b4d419ee33f3127f76439"
format_stamp: "Formatted at 2026-09-26 05:46:42 on dist-test-slave-21p4"
server_key: "e1ddf74600cb739cb9ded538b0c2ad0a"
server_key_iv: "30b1ab0fa22d92733751b8242e3d20e4"
server_key_version: "encryptionkey@0"
I20260926 05:46:42.962229 13436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:42.986878 13436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:42.989166 13436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:42.989347 13436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:42.989895 13436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:42.989959 13436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:42.990026 13436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:42.990077 13436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:46:42 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401602, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:42.997500 13436 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:35317
I20260926 05:46:42.997578 13566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:35317 every 8 connection(s)
I20260926 05:46:42.997956 13436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:43.000130 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:42.990447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:46343 (local address 127.11.247.62:33335)
0926 05:46:42.990493 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:42.990496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:42.990790 (+ 294us) server_negotiation.cc:407] Connection header received
0926 05:46:42.991374 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:42.991375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:42.991392 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:42.991455 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:42.992161 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:42.992697 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:42.994026 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:42.994396 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:42.994484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:42.996937 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:42.996939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:42.996939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:42.996949 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:42.998713 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:42.999123 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:42.999124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:42.999125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:42.999184 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:42.999448 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:42.999449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:42.999449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:42.999580 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:42.999648 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:42.999939 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.000034 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:46:43.000370 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:42.990445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.11.247.1:46343)
0926 05:46:42.990658 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:42.990707 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:42.991265 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:42.991536 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:42.991545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:42.991850 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:42.992053 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:42.992058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:42.992804 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:42.992805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:42.993956 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:42.993959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:42.994486 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:42.994487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:42.994671 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:42.995330 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:42.995339 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:42.996824 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:42.998799 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:42.998803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:42.998806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:42.999043 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:42.999258 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:42.999259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:42.999259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:42.999378 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:42.999650 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:42.999658 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:42.999804 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:46:42.999906 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.000087 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260926 05:46:43.000873 13567 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33335
I20260926 05:46:43.000962 13567 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:43.001165 13567 heartbeater.cc:511] Master 127.11.247.62:33335 requested a full tablet report, sending...
I20260926 05:46:43.001731 13363 ts_manager.cc:194] Registered new tserver with Master: 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317)
I20260926 05:46:43.002728 13363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:46343
I20260926 05:46:43.003520 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13436
I20260926 05:46:43.003656 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:43.003829 12252 external_mini_cluster.cc:1428] Setting key cbf7dd6c2ae159b693f4ff129ae88720
I20260926 05:46:43.008309 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.001942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.11.247.1:47837)
0926 05:46:43.002124 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.002136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.002200 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.002317 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.002318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.002516 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:43.002623 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.002626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.003372 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.003373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.004095 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.004097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.004577 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.004578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.004665 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.005115 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:43.005118 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:43.005399 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:43.007238 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.007239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.007239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.007425 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.007655 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.007656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.007656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.007724 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.008019 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:43.008020 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:43.008070 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.008143 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.008234 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
I20260926 05:46:43.008417 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.001980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:47837 (local address 127.11.247.62:33335)
0926 05:46:43.002039 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.002042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.002200 (+ 158us) server_negotiation.cc:407] Connection header received
0926 05:46:43.002255 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.002256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.002264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.002304 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.002692 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.003288 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.004171 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.004467 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.004541 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.005489 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.005491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.005491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.005499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.007153 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.007516 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.007517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.007518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.007584 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.007785 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.007786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.007786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.007935 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.008004 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.008187 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.008291 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:43.029892 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33335
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:43.115137 13576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:43.115366 13576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:43.115393 13576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:43.116699 13576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:43.116750 13576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:43.116797 13576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:46:43.116837 13576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:43.118389 13576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:33335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:43.118695 13576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:43.118956 13576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:43.124382 13581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:43.124370 13582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:43.124373 13584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:43.124675 13576 server_base.cc:1034] running on GCE node
I20260926 05:46:43.125031 13576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:43.125411 13576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:43.126657 13576 hybrid_clock.cc:648] HybridClock initialized: now 1790401603126641 us; error 38 us; skew 500 ppm
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:43.129307 13576 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:43.129819 13576 webserver.cc:460] Webserver started at http://127.11.247.2:33477/ using document root <none> and password file <none>
I20260926 05:46:43.130082 13576 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:43.130151 13576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:43.130302 13576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:43.131633 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "ee1778851a0e4d5b851ec491cdee27bc"
format_stamp: "Formatted at 2026-09-26 05:46:43 on dist-test-slave-21p4"
server_key: "7fbb97d4ad2a2ed3190b4693c51b9793"
server_key_iv: "e2b18b9bb9e5ca35106f34f7105eea16"
server_key_version: "encryptionkey@0"
I20260926 05:46:43.132009 13576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "ee1778851a0e4d5b851ec491cdee27bc"
format_stamp: "Formatted at 2026-09-26 05:46:43 on dist-test-slave-21p4"
server_key: "7fbb97d4ad2a2ed3190b4693c51b9793"
server_key_iv: "e2b18b9bb9e5ca35106f34f7105eea16"
server_key_version: "encryptionkey@0"
I20260926 05:46:43.133448 13576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:46:43.134343 13591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.134625 13576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:43.134730 13576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "ee1778851a0e4d5b851ec491cdee27bc"
format_stamp: "Formatted at 2026-09-26 05:46:43 on dist-test-slave-21p4"
server_key: "7fbb97d4ad2a2ed3190b4693c51b9793"
server_key_iv: "e2b18b9bb9e5ca35106f34f7105eea16"
server_key_version: "encryptionkey@0"
I20260926 05:46:43.134807 13576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:43.148046 13576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:43.150180 13576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:43.150326 13576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:43.150945 13576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:43.151013 13576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.151059 13576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:43.151093 13576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:43.158404 13576 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:40657
I20260926 05:46:43.158486 13706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:40657 every 8 connection(s)
I20260926 05:46:43.158839 13576 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:43.161312 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.151617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:40113 (local address 127.11.247.62:33335)
0926 05:46:43.151673 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.151675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.151905 (+ 230us) server_negotiation.cc:407] Connection header received
0926 05:46:43.152482 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.152483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.152498 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.152549 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.153432 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.153955 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.155218 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.155418 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.155513 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.158251 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.158253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.158253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.158261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.159939 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.160351 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.160352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.160353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.160415 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.160662 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.160663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.160663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.160790 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.160862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.161134 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.161216 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:46:43.161511 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.151603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.11.247.2:40113)
0926 05:46:43.151788 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.151825 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.152357 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.152683 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.152687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.153041 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:43.153288 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.153294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.154052 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.154053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.155129 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.155131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.155536 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.155537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.155678 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.156261 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:43.156272 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:43.158151 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:43.160034 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.160040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.160043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.160263 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.160477 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.160478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.160478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.160596 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.160853 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:43.160859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:43.161000 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.161110 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.161277 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260926 05:46:43.161993 13707 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33335
I20260926 05:46:43.162089 13707 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:43.162266 13707 heartbeater.cc:511] Master 127.11.247.62:33335 requested a full tablet report, sending...
I20260926 05:46:43.162693 13364 ts_manager.cc:194] Registered new tserver with Master: ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657)
I20260926 05:46:43.163210 13364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:40113
I20260926 05:46:43.165068 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13576
I20260926 05:46:43.165160 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:43.165360 12252 external_mini_cluster.cc:1428] Setting key 5591bdfe870004f933216cb9ef31bdb9
I20260926 05:46:43.169526 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.163121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:52337 (local address 127.11.247.62:33335)
0926 05:46:43.163158 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.163161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.163437 (+ 276us) server_negotiation.cc:407] Connection header received
0926 05:46:43.163518 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.163519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.163531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.163623 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.163998 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.164470 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.165278 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.165535 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.165601 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.166702 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.166703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.166704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.166710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.168367 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.168727 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.168728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.168729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.168786 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.168981 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.168982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.168982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.169132 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.169209 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.169356 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.169434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:46:43.169539 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.163061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.11.247.2:52337)
0926 05:46:43.163386 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.163396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.163447 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.163647 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.163648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.163854 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:43.163919 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.163921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.164560 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.164561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.165201 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.165204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.165599 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.165600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.165703 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.166169 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:43.166176 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:43.166613 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:43.168458 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.168459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.168459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.168631 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.168854 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.168855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.168855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.168914 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.169200 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:43.169201 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:43.169248 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.169330 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.169434 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:43.189878 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33335
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:43.272693 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:43.272917 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:43.272944 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:43.274067 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:43.274102 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:43.274122 13716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:46:43.274140 13716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:43.275595 13716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:33335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:43.275792 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:43.275985 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:43.281111 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:43.281139 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:43.281114 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:43.281404 13716 server_base.cc:1034] running on GCE node
I20260926 05:46:43.281754 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:43.282030 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:43.283195 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790401603283179 us; error 48 us; skew 500 ppm
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:43.285874 13716 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:43.286304 13716 webserver.cc:460] Webserver started at http://127.11.247.3:37335/ using document root <none> and password file <none>
I20260926 05:46:43.286501 13716 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:43.286543 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:43.286720 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:43.287858 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "4d614417cebc450cb0ad6be700ba195c"
format_stamp: "Formatted at 2026-09-26 05:46:43 on dist-test-slave-21p4"
server_key: "14a1f74a23f5a535c20fd27e7ea91a9e"
server_key_iv: "23f2f6714a9354ee7ab2e8e8698aac4d"
server_key_version: "encryptionkey@0"
I20260926 05:46:43.288173 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "4d614417cebc450cb0ad6be700ba195c"
format_stamp: "Formatted at 2026-09-26 05:46:43 on dist-test-slave-21p4"
server_key: "14a1f74a23f5a535c20fd27e7ea91a9e"
server_key_iv: "23f2f6714a9354ee7ab2e8e8698aac4d"
server_key_version: "encryptionkey@0"
I20260926 05:46:43.289520 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:43.290315 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.290496 13716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:43.290614 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "4d614417cebc450cb0ad6be700ba195c"
format_stamp: "Formatted at 2026-09-26 05:46:43 on dist-test-slave-21p4"
server_key: "14a1f74a23f5a535c20fd27e7ea91a9e"
server_key_iv: "23f2f6714a9354ee7ab2e8e8698aac4d"
server_key_version: "encryptionkey@0"
I20260926 05:46:43.290731 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:43.307890 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:43.310349 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:43.310514 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:43.311107 13716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:43.311164 13716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.311192 13716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:43.311234 13716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.318411 13716 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:40633
I20260926 05:46:43.318531 13846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:40633 every 8 connection(s)
I20260926 05:46:43.318974 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:43.322889 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.311627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:51471 (local address 127.11.247.62:33335)
0926 05:46:43.311673 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.311675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.311930 (+ 255us) server_negotiation.cc:407] Connection header received
0926 05:46:43.312470 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.312472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.312486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.312533 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.313578 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.314486 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.316176 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.316514 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.316594 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.319947 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.319950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.319950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.319959 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.321347 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.321800 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.321801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.321802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.321866 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.322121 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.322122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.322122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.322250 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.322323 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.322701 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.322788 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:46:43.323127 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.311633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.11.247.3:51471)
0926 05:46:43.311801 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.311837 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.312355 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.312656 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.312661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.313107 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:43.313446 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.313455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.314674 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.314675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.316073 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.316076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.316655 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.316656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.316787 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.317668 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:43.317690 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:43.319833 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:43.321467 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.321475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.321477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.321708 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.321931 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.321932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.321933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.322054 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.322314 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:43.322320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:43.322486 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.322675 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.322857 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260926 05:46:43.323652 13847 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33335
I20260926 05:46:43.323740 13847 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:43.323967 13847 heartbeater.cc:511] Master 127.11.247.62:33335 requested a full tablet report, sending...
I20260926 05:46:43.324388 13363 ts_manager.cc:194] Registered new tserver with Master: 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633)
I20260926 05:46:43.324831 13363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:51471
I20260926 05:46:43.325237 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13716
I20260926 05:46:43.325320 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:43.325479 12252 external_mini_cluster.cc:1428] Setting key 3e8bdd6009df8f1fe825f854548330b4
I20260926 05:46:43.326332 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:46:43.331907 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.325187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.11.247.3:40005)
0926 05:46:43.325305 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.325316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.325368 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.325892 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.325893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.326149 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:43.326222 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.326225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.327145 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.327146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.327651 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.327652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.328150 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.328151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.328238 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.328696 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:43.328701 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:43.329150 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:43.330819 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.330821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.330821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.331056 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.331306 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.331307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.331307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.331365 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.331652 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:43.331653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:43.331693 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.331744 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.331831 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260926 05:46:43.332000 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.325224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:40005 (local address 127.11.247.62:33335)
0926 05:46:43.325771 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.325772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.325780 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:46:43.325805 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.325806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.325813 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.325886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.326311 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.327032 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.327729 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.328056 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.328133 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.329241 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.329243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.329243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.329249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.330747 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.331161 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.331162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.331163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.331236 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.331428 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.331429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.331430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.331585 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.331663 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.331772 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.331856 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518}
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:43.350374 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.343157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.11.247.62:33335)
0926 05:46:43.343222 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.343223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.343232 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:43.343259 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.343259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.343268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.343332 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.343975 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.344493 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.345287 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.345531 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.345600 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.347976 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.347979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.347979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.347989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.349465 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.349784 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.349785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.349785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.349836 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.349979 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.349979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.349979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.350079 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.350157 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.350241 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.350302 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:46:43.352044 13363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:43.353626 13363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:43.360380 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.356162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:40657 (local address 127.0.0.1:60782)
0926 05:46:43.356497 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.356509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.356623 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.356878 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.356883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.356886 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.356959 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.356960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.357922 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.357924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.359142 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.359145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.360014 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.360016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.360111 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.360126 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.360224 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.360285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 05:46:43.360388 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.356253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.11.247.2:40657)
0926 05:46:43.356505 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.356509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.356600 (+ 91us) server_negotiation.cc:407] Connection header received
0926 05:46:43.356707 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.356710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.356725 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.356797 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.357052 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.357840 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.359228 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.359919 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.360004 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.360068 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.360243 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.360284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
I20260926 05:46:43.360754 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.356162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35317 (local address 127.0.0.1:38484)
0926 05:46:43.356426 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.356471 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.356716 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.356992 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.356994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.356996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.357070 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.357072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.358428 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.358429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.359568 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.359571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.360527 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.360528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.360596 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.360598 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.360642 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.360682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":19,"thread_start_us":61,"threads_started":1}
I20260926 05:46:43.361022 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.356591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.11.247.3:40633)
0926 05:46:43.356814 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.356818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.356827 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:43.356867 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.356869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.356885 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.356959 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.357182 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.358313 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.359553 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.360287 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.360360 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.360417 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.360878 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.360929 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260926 05:46:43.361095 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.356262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.11.247.1:35317)
0926 05:46:43.356402 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.356408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.356539 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:46:43.356841 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.356844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.356860 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.356929 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.357151 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.358338 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.359661 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.360383 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.360446 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.360497 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.360931 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.360974 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260926 05:46:43.361212 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.356371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40633 (local address 127.0.0.1:35716)
0926 05:46:43.356591 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.356600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.356646 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.357029 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.357030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.357032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.357087 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.357089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.358413 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.358414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.359473 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.359474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.360455 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.360456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.360521 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.360526 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.360586 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.361144 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":31,"thread_start_us":84,"threads_started":1}
I20260926 05:46:43.361384 13641 tablet_service.cc:1467] Processing CreateTablet for tablet 09e909aec8334f71b15286ee19c0bdbf (DEFAULT_TABLE table=test-table [id=0ed3211a782249ccbb351f2fdb5c9b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:43.361791 13641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e909aec8334f71b15286ee19c0bdbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.361690 13501 tablet_service.cc:1467] Processing CreateTablet for tablet 09e909aec8334f71b15286ee19c0bdbf (DEFAULT_TABLE table=test-table [id=0ed3211a782249ccbb351f2fdb5c9b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:43.361770 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 09e909aec8334f71b15286ee19c0bdbf (DEFAULT_TABLE table=test-table [id=0ed3211a782249ccbb351f2fdb5c9b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:43.362053 13501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e909aec8334f71b15286ee19c0bdbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.362053 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e909aec8334f71b15286ee19c0bdbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.365235 13870 tablet_bootstrap.cc:492] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Bootstrap starting.
I20260926 05:46:43.365687 13871 tablet_bootstrap.cc:492] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Bootstrap starting.
I20260926 05:46:43.365851 13872 tablet_bootstrap.cc:492] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Bootstrap starting.
I20260926 05:46:43.365880 13870 tablet_bootstrap.cc:654] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.366235 13870 log.cc:824] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:43.366652 13872 tablet_bootstrap.cc:654] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.366672 13871 tablet_bootstrap.cc:654] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.366922 13872 log.cc:824] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:43.366989 13871 log.cc:824] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:43.367072 13870 tablet_bootstrap.cc:492] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: No bootstrap required, opened a new log
I20260926 05:46:43.367132 13870 ts_tablet_manager.cc:1397] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:43.367631 13872 tablet_bootstrap.cc:492] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: No bootstrap required, opened a new log
I20260926 05:46:43.367690 13872 ts_tablet_manager.cc:1397] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:46:43.367796 13871 tablet_bootstrap.cc:492] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: No bootstrap required, opened a new log
I20260926 05:46:43.367844 13871 ts_tablet_manager.cc:1397] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 05:46:43.368685 13870 raft_consensus.cc:348] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.368808 13870 raft_consensus.cc:374] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.368829 13870 raft_consensus.cc:729] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1778851a0e4d5b851ec491cdee27bc, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.368945 13870 consensus_queue.cc:260] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.369208 13870 ts_tablet_manager.cc:1428] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:43.369218 13707 heartbeater.cc:503] Master 127.11.247.62:33335 was elected leader, sending a full tablet report...
I20260926 05:46:43.369705 13872 raft_consensus.cc:348] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.369777 13871 raft_consensus.cc:348] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.369875 13872 raft_consensus.cc:374] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.369899 13871 raft_consensus.cc:374] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.369931 13871 raft_consensus.cc:729] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d614417cebc450cb0ad6be700ba195c, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.369915 13872 raft_consensus.cc:729] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82652eaeb41b4d419ee33f3127f76439, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.370038 13872 consensus_queue.cc:260] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.370054 13871 consensus_queue.cc:260] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.370303 13871 ts_tablet_manager.cc:1428] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:46:43.370303 13872 ts_tablet_manager.cc:1428] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:46:43.370402 13567 heartbeater.cc:503] Master 127.11.247.62:33335 was elected leader, sending a full tablet report...
I20260926 05:46:43.370677 13847 heartbeater.cc:503] Master 127.11.247.62:33335 was elected leader, sending a full tablet report...
W20260926 05:46:43.409461 13708 tablet.cc:2369] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:46:43.498787 13568 tablet.cc:2369] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:46:43.557009 13877 raft_consensus.cc:482] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:43.557148 13877 raft_consensus.cc:504] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.557598 13877 leader_election.cc:288] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657), 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317)
I20260926 05:46:43.561276 13878 raft_consensus.cc:482] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:43.561494 13878 raft_consensus.cc:504] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.561683 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.557792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:40657 (local address 127.11.247.3:46289)
0926 05:46:43.557867 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.557879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.557928 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.558147 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.558148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.558150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.558226 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.558228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.559207 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.559208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.560604 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.560607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.561377 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.561378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.561467 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.561472 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.561536 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.561608 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":66}
I20260926 05:46:43.561689 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.557857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:46289 (local address 127.11.247.2:40657)
0926 05:46:43.557905 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.557908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.557920 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:46:43.558041 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.558042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.558064 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.558119 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.558297 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.559099 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.560709 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.561300 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.561376 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.561385 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.561566 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.561619 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:46:43.561741 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.557927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:53121 (local address 127.11.247.1:35317)
0926 05:46:43.557973 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.557974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.558075 (+ 101us) server_negotiation.cc:407] Connection header received
0926 05:46:43.558206 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.558206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.558214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.558257 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.558391 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.559109 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.560708 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.561294 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.561376 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.561385 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.561624 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.561673 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:46:43.561949 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.557832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35317 (local address 127.11.247.3:53121)
0926 05:46:43.557999 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.558014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.558070 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.558273 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.558274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.558275 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.558335 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.558336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.559207 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.559209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.560603 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.560607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.561441 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.561443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.561528 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.561533 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.561809 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.561858 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":72,"thread_start_us":118,"threads_started":1}
I20260926 05:46:43.562280 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e909aec8334f71b15286ee19c0bdbf" candidate_uuid: "4d614417cebc450cb0ad6be700ba195c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1778851a0e4d5b851ec491cdee27bc" is_pre_election: true
I20260926 05:46:43.562461 13661 raft_consensus.cc:2454] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d614417cebc450cb0ad6be700ba195c in term 0.
I20260926 05:46:43.562629 13878 leader_election.cc:288] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657), 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633)
I20260926 05:46:43.562713 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e909aec8334f71b15286ee19c0bdbf" candidate_uuid: "4d614417cebc450cb0ad6be700ba195c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82652eaeb41b4d419ee33f3127f76439" is_pre_election: true
I20260926 05:46:43.562789 13521 raft_consensus.cc:2454] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d614417cebc450cb0ad6be700ba195c in term 0.
I20260926 05:46:43.562793 13734 leader_election.cc:302] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d614417cebc450cb0ad6be700ba195c, ee1778851a0e4d5b851ec491cdee27bc; no voters:
I20260926 05:46:43.562949 13877 raft_consensus.cc:2781] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:43.563095 13877 raft_consensus.cc:482] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:43.563129 13877 raft_consensus.cc:3037] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.564107 13877 raft_consensus.cc:504] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.564404 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e909aec8334f71b15286ee19c0bdbf" candidate_uuid: "4d614417cebc450cb0ad6be700ba195c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1778851a0e4d5b851ec491cdee27bc"
I20260926 05:46:43.564522 13661 raft_consensus.cc:3037] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.564509 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e909aec8334f71b15286ee19c0bdbf" candidate_uuid: "4d614417cebc450cb0ad6be700ba195c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82652eaeb41b4d419ee33f3127f76439"
I20260926 05:46:43.564622 13521 raft_consensus.cc:3037] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.565500 13661 raft_consensus.cc:2454] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d614417cebc450cb0ad6be700ba195c in term 1.
I20260926 05:46:43.565503 13521 raft_consensus.cc:2454] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d614417cebc450cb0ad6be700ba195c in term 1.
I20260926 05:46:43.565721 13732 leader_election.cc:302] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d614417cebc450cb0ad6be700ba195c, 82652eaeb41b4d419ee33f3127f76439; no voters:
I20260926 05:46:43.565934 13877 leader_election.cc:288] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [CANDIDATE]: Term 1 election: Requested vote from peers ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657), 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317)
I20260926 05:46:43.565940 13882 raft_consensus.cc:2781] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:43.566098 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.562378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:35245 (local address 127.11.247.2:40657)
0926 05:46:43.562416 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.562418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.562558 (+ 140us) server_negotiation.cc:407] Connection header received
0926 05:46:43.562677 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.562678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.562689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.562719 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.562909 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.563602 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.565058 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.565630 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.565677 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.565688 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.565978 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.566024 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:46:43.566160 13882 raft_consensus.cc:686] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 1 LEADER]: Becoming Leader. State: Replica: 4d614417cebc450cb0ad6be700ba195c, State: Running, Role: LEADER
I20260926 05:46:43.566287 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.562326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:40657 (local address 127.11.247.1:35245)
0926 05:46:43.562492 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.562504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.562583 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.562779 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.562780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.562783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.562831 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.562833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.563734 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.563735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.564940 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.564943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.565807 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.565808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.565888 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.565892 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.566160 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.566208 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260926 05:46:43.566267 13882 consensus_queue.cc:237] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } }
I20260926 05:46:43.566563 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e909aec8334f71b15286ee19c0bdbf" candidate_uuid: "82652eaeb41b4d419ee33f3127f76439" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1778851a0e4d5b851ec491cdee27bc" is_pre_election: true
I20260926 05:46:43.566704 13661 raft_consensus.cc:2379] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82652eaeb41b4d419ee33f3127f76439 in current term 1: Already voted for candidate 4d614417cebc450cb0ad6be700ba195c in this term.
I20260926 05:46:43.566937 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.563027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:36099 (local address 127.11.247.3:40633)
0926 05:46:43.563072 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.563074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.563191 (+ 117us) server_negotiation.cc:407] Connection header received
0926 05:46:43.563293 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.563294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.563303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.563374 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.563518 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.564411 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.565918 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.566512 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.566654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.566663 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.566840 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.566882 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:46:43.567137 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.562989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40633 (local address 127.11.247.1:36099)
0926 05:46:43.563136 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.563145 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.563214 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.563377 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.563378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.563379 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.563445 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.563447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.564524 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.564525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.565807 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.565810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.566655 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.566656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.566736 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.566740 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.567009 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.567065 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
I20260926 05:46:43.567415 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e909aec8334f71b15286ee19c0bdbf" candidate_uuid: "82652eaeb41b4d419ee33f3127f76439" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d614417cebc450cb0ad6be700ba195c" is_pre_election: true
I20260926 05:46:43.567539 13363 catalog_manager.cc:5495] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d614417cebc450cb0ad6be700ba195c (127.11.247.3). New cstate: current_term: 1 leader_uuid: "4d614417cebc450cb0ad6be700ba195c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } health_report { overall_health: UNKNOWN } } }
I20260926 05:46:43.567699 13454 leader_election.cc:302] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82652eaeb41b4d419ee33f3127f76439; no voters: 4d614417cebc450cb0ad6be700ba195c, ee1778851a0e4d5b851ec491cdee27bc
I20260926 05:46:43.567850 13878 raft_consensus.cc:2726] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 05:46:43.569594 13848 tablet.cc:2369] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:46:43.587751 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.585130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.11.247.3:40633)
0926 05:46:43.585192 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.585194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.585225 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:46:43.585260 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.585261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.585278 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.585330 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:43.585568 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.586075 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.586934 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.587136 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.587197 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.587372 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.587485 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.587546 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.587603 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.587622 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:46:43.589757 13521 raft_consensus.cc:1261] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Refusing update from remote peer 4d614417cebc450cb0ad6be700ba195c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:43.589769 13661 raft_consensus.cc:1261] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Refusing update from remote peer 4d614417cebc450cb0ad6be700ba195c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:43.590103 13882 consensus_queue.cc:1035] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:43.590224 13877 consensus_queue.cc:1035] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:43.592751 13885 mvcc.cc:205] Tried to move back new op lower bound from 7333484968300572672 to 7333484968208347136. Current Snapshot: MvccSnapshot[applied={T|T < 7333484968300572672}]
I20260926 05:46:43.597152 13363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:43.597265 13363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:43.598021 13363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: Sending DeleteTablet for 3 replicas of tablet 09e909aec8334f71b15286ee19c0bdbf
I20260926 05:46:43.598488 13501 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e909aec8334f71b15286ee19c0bdbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38484
I20260926 05:46:43.598616 13641 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e909aec8334f71b15286ee19c0bdbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60782
I20260926 05:46:43.598798 13781 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e909aec8334f71b15286ee19c0bdbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:35716
I20260926 05:46:43.598850 13895 tablet_replica.cc:330] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: stopping tablet replica
I20260926 05:46:43.599038 13895 raft_consensus.cc:2229] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:43.599090 13363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:43.599262 13363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:43.599197 13895 raft_consensus.cc:2258] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:43.599574 13896 tablet_replica.cc:330] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: stopping tablet replica
I20260926 05:46:43.599022 13894 tablet_replica.cc:330] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: stopping tablet replica
I20260926 05:46:43.599920 13895 ts_tablet_manager.cc:1905] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:43.600414 13896 raft_consensus.cc:2229] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:46:43.600512 13781 tablet_service.cc:1467] Processing CreateTablet for tablet 4c38277aaa3c4e6e8e098bec20a85f4d (DEFAULT_TABLE table=test-table [id=1fc0b2fe52124472a74e67ee4a612ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:43.600544 13896 raft_consensus.cc:2258] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:43.600643 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c38277aaa3c4e6e8e098bec20a85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.600710 13894 raft_consensus.cc:2229] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:43.600833 13894 raft_consensus.cc:2258] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:43.601439 13896 ts_tablet_manager.cc:1905] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:43.602177 13871 tablet_bootstrap.cc:492] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Bootstrap starting.
I20260926 05:46:43.602344 13894 ts_tablet_manager.cc:1905] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:43.602424 13501 tablet_service.cc:1467] Processing CreateTablet for tablet 4c38277aaa3c4e6e8e098bec20a85f4d (DEFAULT_TABLE table=test-table [id=1fc0b2fe52124472a74e67ee4a612ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:43.602514 13501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c38277aaa3c4e6e8e098bec20a85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.602655 13871 tablet_bootstrap.cc:654] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.603124 13895 ts_tablet_manager.cc:1918] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:46:43.603243 13895 log.cc:1196] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/09e909aec8334f71b15286ee19c0bdbf
I20260926 05:46:43.603600 13895 ts_tablet_manager.cc:1939] T 09e909aec8334f71b15286ee19c0bdbf P ee1778851a0e4d5b851ec491cdee27bc: Deleting consensus metadata
I20260926 05:46:43.603868 13872 tablet_bootstrap.cc:492] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Bootstrap starting.
I20260926 05:46:43.604182 13350 catalog_manager.cc:4841] TS ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657): tablet 09e909aec8334f71b15286ee19c0bdbf (table test-table [id=0ed3211a782249ccbb351f2fdb5c9b31]) successfully deleted
I20260926 05:46:43.604571 13872 tablet_bootstrap.cc:654] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.605118 13641 tablet_service.cc:1467] Processing CreateTablet for tablet 4c38277aaa3c4e6e8e098bec20a85f4d (DEFAULT_TABLE table=test-table [id=1fc0b2fe52124472a74e67ee4a612ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:43.605229 13872 tablet_bootstrap.cc:492] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: No bootstrap required, opened a new log
I20260926 05:46:43.605238 13641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c38277aaa3c4e6e8e098bec20a85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.605273 13872 ts_tablet_manager.cc:1397] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:43.605427 13872 raft_consensus.cc:348] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.605495 13872 raft_consensus.cc:374] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.605516 13872 raft_consensus.cc:729] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82652eaeb41b4d419ee33f3127f76439, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.605574 13896 ts_tablet_manager.cc:1918] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:46:43.605638 13896 log.cc:1196] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/09e909aec8334f71b15286ee19c0bdbf
I20260926 05:46:43.605572 13872 consensus_queue.cc:260] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.605737 13872 ts_tablet_manager.cc:1428] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.606602 13870 tablet_bootstrap.cc:492] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Bootstrap starting.
I20260926 05:46:43.606868 13896 ts_tablet_manager.cc:1939] T 09e909aec8334f71b15286ee19c0bdbf P 4d614417cebc450cb0ad6be700ba195c: Deleting consensus metadata
I20260926 05:46:43.606978 13870 tablet_bootstrap.cc:654] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.605276 13894 ts_tablet_manager.cc:1918] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:46:43.607322 13894 log.cc:1196] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/09e909aec8334f71b15286ee19c0bdbf
I20260926 05:46:43.607342 13350 catalog_manager.cc:4841] TS 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633): tablet 09e909aec8334f71b15286ee19c0bdbf (table test-table [id=0ed3211a782249ccbb351f2fdb5c9b31]) successfully deleted
I20260926 05:46:43.607625 13894 ts_tablet_manager.cc:1939] T 09e909aec8334f71b15286ee19c0bdbf P 82652eaeb41b4d419ee33f3127f76439: Deleting consensus metadata
I20260926 05:46:43.607832 13870 tablet_bootstrap.cc:492] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: No bootstrap required, opened a new log
I20260926 05:46:43.607882 13870 ts_tablet_manager.cc:1397] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:43.608053 13870 raft_consensus.cc:348] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.608141 13870 raft_consensus.cc:374] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.608163 13870 raft_consensus.cc:729] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1778851a0e4d5b851ec491cdee27bc, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.608206 13870 consensus_queue.cc:260] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.608291 13870 ts_tablet_manager.cc:1428] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.608335 13871 tablet_bootstrap.cc:492] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: No bootstrap required, opened a new log
I20260926 05:46:43.608392 13871 ts_tablet_manager.cc:1397] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260926 05:46:43.608428 13348 catalog_manager.cc:4841] TS 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317): tablet 09e909aec8334f71b15286ee19c0bdbf (table test-table [id=0ed3211a782249ccbb351f2fdb5c9b31]) successfully deleted
I20260926 05:46:43.608538 13871 raft_consensus.cc:348] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.608608 13871 raft_consensus.cc:374] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.608628 13871 raft_consensus.cc:729] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d614417cebc450cb0ad6be700ba195c, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.608675 13871 consensus_queue.cc:260] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.608774 13871 ts_tablet_manager.cc:1428] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:43.745925 13876 raft_consensus.cc:482] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:43.746034 13876 raft_consensus.cc:504] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.746481 13876 leader_election.cc:288] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317), 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633)
I20260926 05:46:43.749655 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.746725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:51567 (local address 127.11.247.3:40633)
0926 05:46:43.746773 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.746776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.746916 (+ 140us) server_negotiation.cc:407] Connection header received
0926 05:46:43.747047 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.747048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.747059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.747106 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.747268 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.747799 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.748923 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.749289 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.749372 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.749381 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.749542 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.749582 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:46:43.749665 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.746819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:42943 (local address 127.11.247.1:35317)
0926 05:46:43.746893 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.746896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.746972 (+ 76us) server_negotiation.cc:407] Connection header received
0926 05:46:43.747100 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.747101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.747111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.747179 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:43.747327 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.747795 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.748897 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.749337 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.749406 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.749415 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.749555 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.749598 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:46:43.749681 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.746679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40633 (local address 127.11.247.2:51567)
0926 05:46:43.746831 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.746857 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.746971 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.747111 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.747112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.747114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.747194 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.747196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.747894 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.747895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.748828 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.748830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.749372 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.749373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.749457 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.749460 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.749533 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.749579 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":33,"thread_start_us":73,"threads_started":1}
I20260926 05:46:43.749687 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.746691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35317 (local address 127.11.247.2:42943)
0926 05:46:43.746910 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.746920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.747002 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.747182 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.747183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.747185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:43.747255 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.747257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.747926 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.747927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.748798 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.748800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.749406 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.749407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.749483 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.749486 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.749550 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.749592 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":15,"thread_start_us":58,"threads_started":1}
I20260926 05:46:43.750000 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c38277aaa3c4e6e8e098bec20a85f4d" candidate_uuid: "ee1778851a0e4d5b851ec491cdee27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d614417cebc450cb0ad6be700ba195c" is_pre_election: true
I20260926 05:46:43.750129 13801 raft_consensus.cc:2454] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee1778851a0e4d5b851ec491cdee27bc in term 0.
I20260926 05:46:43.750188 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c38277aaa3c4e6e8e098bec20a85f4d" candidate_uuid: "ee1778851a0e4d5b851ec491cdee27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82652eaeb41b4d419ee33f3127f76439" is_pre_election: true
I20260926 05:46:43.750279 13521 raft_consensus.cc:2454] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee1778851a0e4d5b851ec491cdee27bc in term 0.
I20260926 05:46:43.750298 13594 leader_election.cc:302] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d614417cebc450cb0ad6be700ba195c, ee1778851a0e4d5b851ec491cdee27bc; no voters:
I20260926 05:46:43.750422 13876 raft_consensus.cc:2781] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:43.750458 13876 raft_consensus.cc:482] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:43.750480 13876 raft_consensus.cc:3037] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.751238 13876 raft_consensus.cc:504] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.751389 13876 leader_election.cc:288] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [CANDIDATE]: Term 1 election: Requested vote from peers 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317), 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633)
I20260926 05:46:43.751545 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c38277aaa3c4e6e8e098bec20a85f4d" candidate_uuid: "ee1778851a0e4d5b851ec491cdee27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82652eaeb41b4d419ee33f3127f76439"
I20260926 05:46:43.751616 13521 raft_consensus.cc:3037] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.752185 13521 raft_consensus.cc:2454] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee1778851a0e4d5b851ec491cdee27bc in term 1.
I20260926 05:46:43.752368 13592 leader_election.cc:302] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82652eaeb41b4d419ee33f3127f76439, ee1778851a0e4d5b851ec491cdee27bc; no voters:
I20260926 05:46:43.752374 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c38277aaa3c4e6e8e098bec20a85f4d" candidate_uuid: "ee1778851a0e4d5b851ec491cdee27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d614417cebc450cb0ad6be700ba195c"
I20260926 05:46:43.752446 13801 raft_consensus.cc:3037] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.752532 13876 raft_consensus.cc:2781] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:43.752668 13876 raft_consensus.cc:686] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 1 LEADER]: Becoming Leader. State: Replica: ee1778851a0e4d5b851ec491cdee27bc, State: Running, Role: LEADER
I20260926 05:46:43.752745 13876 consensus_queue.cc:237] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.753324 13801 raft_consensus.cc:2454] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee1778851a0e4d5b851ec491cdee27bc in term 1.
I20260926 05:46:43.753505 13363 catalog_manager.cc:5495] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2). New cstate: current_term: 1 leader_uuid: "ee1778851a0e4d5b851ec491cdee27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } health_report { overall_health: UNKNOWN } } }
Sep 26 05:46:43 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401602, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:43.806907 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.799418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33335 (local address 127.0.0.1:58882)
0926 05:46:43.799654 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.799665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.799759 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.799919 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.799920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.800339 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:43.800404 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.800408 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.801076 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.801077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.801675 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.801677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.802014 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.802015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.802066 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.802630 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:43.802635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:43.804112 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:43.805709 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.805717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.805719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.805936 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.806153 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:43.806154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:43.806155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:43.806228 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:43.806539 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:43.806546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:43.806666 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.806726 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.806828 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
I20260926 05:46:43.806972 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.799524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.11.247.62:33335)
0926 05:46:43.799649 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.799651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.799737 (+ 86us) server_negotiation.cc:407] Connection header received
0926 05:46:43.799838 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.799839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.799854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.799914 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:43.800487 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.800992 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.801766 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.801947 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.802015 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.804212 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:43.804214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:43.804215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:43.804229 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:43.805618 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.806005 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.806006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.806006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.806065 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:43.806294 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:43.806295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:43.806295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:43.806456 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:43.806519 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.806763 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.806883 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":61}
I20260926 05:46:43.809206 13363 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:58882:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:46:43.809489 13363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:43.810982 13501 tablet_service.cc:1467] Processing CreateTablet for tablet fe86217c05724f86bd524d6b098a025f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3075680c5be4fd9ac68c7360ef54c62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:46:43.811002 13641 tablet_service.cc:1467] Processing CreateTablet for tablet fe86217c05724f86bd524d6b098a025f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3075680c5be4fd9ac68c7360ef54c62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:46:43.811028 13781 tablet_service.cc:1467] Processing CreateTablet for tablet fe86217c05724f86bd524d6b098a025f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3075680c5be4fd9ac68c7360ef54c62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:46:43.811179 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe86217c05724f86bd524d6b098a025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.811179 13501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe86217c05724f86bd524d6b098a025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.811178 13641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe86217c05724f86bd524d6b098a025f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:43.812467 13872 tablet_bootstrap.cc:492] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: Bootstrap starting.
I20260926 05:46:43.812700 13871 tablet_bootstrap.cc:492] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: Bootstrap starting.
I20260926 05:46:43.812708 13870 tablet_bootstrap.cc:492] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: Bootstrap starting.
I20260926 05:46:43.812958 13872 tablet_bootstrap.cc:654] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.813030 13871 tablet_bootstrap.cc:654] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.813086 13870 tablet_bootstrap.cc:654] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:43.813670 13871 tablet_bootstrap.cc:492] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: No bootstrap required, opened a new log
I20260926 05:46:43.813726 13871 ts_tablet_manager.cc:1397] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:43.813809 13870 tablet_bootstrap.cc:492] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: No bootstrap required, opened a new log
I20260926 05:46:43.813824 13872 tablet_bootstrap.cc:492] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: No bootstrap required, opened a new log
I20260926 05:46:43.813848 13870 ts_tablet_manager.cc:1397] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:43.813864 13872 ts_tablet_manager.cc:1397] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:43.813843 13871 raft_consensus.cc:348] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.813921 13871 raft_consensus.cc:374] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.813943 13871 raft_consensus.cc:729] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d614417cebc450cb0ad6be700ba195c, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.813992 13871 consensus_queue.cc:260] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.813978 13872 raft_consensus.cc:348] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.814065 13872 raft_consensus.cc:374] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.814009 13870 raft_consensus.cc:348] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.814090 13872 raft_consensus.cc:729] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82652eaeb41b4d419ee33f3127f76439, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.814092 13871 ts_tablet_manager.cc:1428] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.814098 13870 raft_consensus.cc:374] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:43.814121 13870 raft_consensus.cc:729] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1778851a0e4d5b851ec491cdee27bc, State: Initialized, Role: FOLLOWER
I20260926 05:46:43.814131 13872 consensus_queue.cc:260] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.814119 13877 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.814167 13870 consensus_queue.cc:260] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.814201 13877 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.814218 13872 ts_tablet_manager.cc:1428] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:43.814260 13870 ts_tablet_manager.cc:1428] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:43.814358 13878 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.814448 13878 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.814620 13876 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.814725 13876 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.925817 13878 raft_consensus.cc:482] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:43.925935 13878 raft_consensus.cc:504] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.926097 13878 leader_election.cc:288] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657), 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633)
I20260926 05:46:43.926388 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86217c05724f86bd524d6b098a025f" candidate_uuid: "82652eaeb41b4d419ee33f3127f76439" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1778851a0e4d5b851ec491cdee27bc" is_pre_election: true
I20260926 05:46:43.926497 13661 raft_consensus.cc:2454] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82652eaeb41b4d419ee33f3127f76439 in term 0.
I20260926 05:46:43.926470 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86217c05724f86bd524d6b098a025f" candidate_uuid: "82652eaeb41b4d419ee33f3127f76439" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d614417cebc450cb0ad6be700ba195c" is_pre_election: true
I20260926 05:46:43.926661 13801 raft_consensus.cc:2454] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82652eaeb41b4d419ee33f3127f76439 in term 0.
I20260926 05:46:43.926718 13454 leader_election.cc:302] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82652eaeb41b4d419ee33f3127f76439, ee1778851a0e4d5b851ec491cdee27bc; no voters:
I20260926 05:46:43.926811 13878 raft_consensus.cc:2781] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:43.926853 13878 raft_consensus.cc:482] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:43.926873 13878 raft_consensus.cc:3037] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.927706 13878 raft_consensus.cc:504] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.927821 13878 leader_election.cc:288] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [CANDIDATE]: Term 1 election: Requested vote from peers ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657), 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633)
I20260926 05:46:43.928011 13661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86217c05724f86bd524d6b098a025f" candidate_uuid: "82652eaeb41b4d419ee33f3127f76439" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee1778851a0e4d5b851ec491cdee27bc"
I20260926 05:46:43.928005 13801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe86217c05724f86bd524d6b098a025f" candidate_uuid: "82652eaeb41b4d419ee33f3127f76439" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d614417cebc450cb0ad6be700ba195c"
I20260926 05:46:43.928079 13661 raft_consensus.cc:3037] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.928093 13801 raft_consensus.cc:3037] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:43.928674 13661 raft_consensus.cc:2454] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82652eaeb41b4d419ee33f3127f76439 in term 1.
I20260926 05:46:43.928855 13801 raft_consensus.cc:2454] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82652eaeb41b4d419ee33f3127f76439 in term 1.
I20260926 05:46:43.928891 13454 leader_election.cc:302] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82652eaeb41b4d419ee33f3127f76439, ee1778851a0e4d5b851ec491cdee27bc; no voters:
I20260926 05:46:43.928978 13878 raft_consensus.cc:2781] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:43.929123 13878 raft_consensus.cc:686] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [term 1 LEADER]: Becoming Leader. State: Replica: 82652eaeb41b4d419ee33f3127f76439, State: Running, Role: LEADER
I20260926 05:46:43.929208 13878 consensus_queue.cc:237] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } }
I20260926 05:46:43.929563 13913 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: TxnStatusTablet state changed. Reason: New leader 82652eaeb41b4d419ee33f3127f76439. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.929680 13913 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:46:43.929939 13915 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:46:43.929940 13363 catalog_manager.cc:5495] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } health_report { overall_health: UNKNOWN } } }
I20260926 05:46:43.930069 13915 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:46:43.961995 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.958923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35317 (local address 127.0.0.1:38490)
0926 05:46:43.959025 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.959038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.959094 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.959277 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.959277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.959278 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:43.959330 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.959332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.960280 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.960281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.960883 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.960885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.961364 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.961364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.961410 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.961428 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.961840 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.961848 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.961875 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.961926 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 05:46:43.962028 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.959006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.11.247.1:35317)
0926 05:46:43.959059 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.959061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.959073 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:46:43.959168 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.959169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.959192 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.959267 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:43.959373 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.960164 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.960967 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.961280 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.961354 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.961613 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.961759 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.961866 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.961907 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.961930 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:46:43.963359 13661 raft_consensus.cc:1261] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Refusing update from remote peer 82652eaeb41b4d419ee33f3127f76439: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:43.963415 13801 raft_consensus.cc:1261] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Refusing update from remote peer 82652eaeb41b4d419ee33f3127f76439: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:43.963693 13913 consensus_queue.cc:1035] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:43.963845 13878 consensus_queue.cc:1035] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:43.964511 13876 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: TxnStatusTablet state changed. Reason: New leader 82652eaeb41b4d419ee33f3127f76439. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.964615 13876 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.964547 13877 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: TxnStatusTablet state changed. Reason: New leader 82652eaeb41b4d419ee33f3127f76439. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.964659 13877 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.965679 13878 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.965801 13878 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:46:43.965972 13887 mvcc.cc:205] Tried to move back new op lower bound from 7333484969831587840 to 7333484969694584832. Current Snapshot: MvccSnapshot[applied={T|T < 7333484969831587840}]
I20260926 05:46:43.966197 13876 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.966286 13876 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P ee1778851a0e4d5b851ec491cdee27bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.966531 13878 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.966636 13878 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 82652eaeb41b4d419ee33f3127f76439: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:46:43.967300 13877 tablet_replica.cc:439] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82652eaeb41b4d419ee33f3127f76439" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 } } peers { permanent_uuid: "ee1778851a0e4d5b851ec491cdee27bc" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 40657 } } peers { permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 } } }
I20260926 05:46:43.967401 13877 tablet_replica.cc:442] T fe86217c05724f86bd524d6b098a025f P 4d614417cebc450cb0ad6be700ba195c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:43.970956 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.967987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.11.247.2:40657)
0926 05:46:43.968025 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.968027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.968033 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:46:43.968067 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.968068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.968088 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.968157 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:43.968283 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.969037 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.969990 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.970301 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.970373 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.970563 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.970727 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.970804 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.970845 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.970865 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:46:43.976719 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.973771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35317 (local address 127.11.247.2:40041)
0926 05:46:43.973947 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.973957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.974012 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.974152 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.974153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.974154 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:43.974211 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.974212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.975154 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.975155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.975841 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.975844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.976330 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.976330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.976377 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.976394 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.976571 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.976573 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.976609 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.976646 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260926 05:46:43.976719 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.973857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:40041 (local address 127.11.247.1:35317)
0926 05:46:43.973945 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.973948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.974007 (+ 59us) server_negotiation.cc:407] Connection header received
0926 05:46:43.974068 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.974069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.974080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.974135 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:43.974269 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.975072 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.975933 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.976238 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.976310 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.976475 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.976511 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.976564 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.976618 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.976638 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 05:46:43.979501 13801 raft_consensus.cc:1261] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Refusing update from remote peer ee1778851a0e4d5b851ec491cdee27bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:43.979552 13521 raft_consensus.cc:1261] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Refusing update from remote peer ee1778851a0e4d5b851ec491cdee27bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:43.979801 13902 consensus_queue.cc:1035] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d614417cebc450cb0ad6be700ba195c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:43.979918 13876 consensus_queue.cc:1035] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "82652eaeb41b4d419ee33f3127f76439" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:43.981540 13886 mvcc.cc:205] Tried to move back new op lower bound from 7333484969898237952 to 7333484968971816960. Current Snapshot: MvccSnapshot[applied={T|T < 7333484969898237952}]
I20260926 05:46:43.989158 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.986299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:34057 (local address 127.11.247.2:40657)
0926 05:46:43.986356 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:43.986358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:43.986517 (+ 159us) server_negotiation.cc:407] Connection header received
0926 05:46:43.986619 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:43.986620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:43.986630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:43.986690 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:43.986832 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.987297 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.988450 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.988663 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.988722 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.988936 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.988965 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.989002 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:43.989080 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:46:43.989098 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:46:43.989410 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:43.986220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:40657 (local address 127.11.247.1:34057)
0926 05:46:43.986450 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:43.986461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:43.986526 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:43.986697 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:43.986698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:43.986699 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:43.986773 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.986775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.987372 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.987373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.988359 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:43.988362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:43.988739 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:43.988740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:43.988811 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:43.988850 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:43.989021 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:43.989024 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:46:43.989078 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 05:46:43.989127 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260926 05:46:44.000300 13363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:44.000427 13363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:44.001108 13363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d6b4c21ea44225b65d7c9c976dc1e0: Sending DeleteTablet for 3 replicas of tablet 4c38277aaa3c4e6e8e098bec20a85f4d
I20260926 05:46:44.001341 13501 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c38277aaa3c4e6e8e098bec20a85f4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38484
I20260926 05:46:44.001446 13641 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c38277aaa3c4e6e8e098bec20a85f4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60782
I20260926 05:46:44.001441 13781 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c38277aaa3c4e6e8e098bec20a85f4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:35716
I20260926 05:46:44.001549 13894 tablet_replica.cc:330] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: stopping tablet replica
I20260926 05:46:44.001605 13894 raft_consensus.cc:2229] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:44.001639 13894 raft_consensus.cc:2258] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:44.001703 13895 tablet_replica.cc:330] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: stopping tablet replica
I20260926 05:46:44.001753 13895 raft_consensus.cc:2229] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:46:44.001847 13895 raft_consensus.cc:2258] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:44.001929 13896 tablet_replica.cc:330] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: stopping tablet replica
I20260926 05:46:44.001993 13896 raft_consensus.cc:2229] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:44.002035 13896 raft_consensus.cc:2258] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:44.002400 13896 ts_tablet_manager.cc:1905] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:44.002774 13894 ts_tablet_manager.cc:1905] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:44.004181 13896 ts_tablet_manager.cc:1918] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:44.004235 13896 log.cc:1196] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/4c38277aaa3c4e6e8e098bec20a85f4d
I20260926 05:46:44.004341 13894 ts_tablet_manager.cc:1918] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:44.004384 13894 log.cc:1196] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/4c38277aaa3c4e6e8e098bec20a85f4d
I20260926 05:46:44.004465 13896 ts_tablet_manager.cc:1939] T 4c38277aaa3c4e6e8e098bec20a85f4d P 4d614417cebc450cb0ad6be700ba195c: Deleting consensus metadata
I20260926 05:46:44.004545 13894 ts_tablet_manager.cc:1939] T 4c38277aaa3c4e6e8e098bec20a85f4d P 82652eaeb41b4d419ee33f3127f76439: Deleting consensus metadata
I20260926 05:46:44.004808 13895 ts_tablet_manager.cc:1905] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:44.004851 13350 catalog_manager.cc:4841] TS 4d614417cebc450cb0ad6be700ba195c (127.11.247.3:40633): tablet 4c38277aaa3c4e6e8e098bec20a85f4d (table test-table [id=1fc0b2fe52124472a74e67ee4a612ec6]) successfully deleted
I20260926 05:46:44.004969 13348 catalog_manager.cc:4841] TS 82652eaeb41b4d419ee33f3127f76439 (127.11.247.1:35317): tablet 4c38277aaa3c4e6e8e098bec20a85f4d (table test-table [id=1fc0b2fe52124472a74e67ee4a612ec6]) successfully deleted
I20260926 05:46:44.006175 13895 ts_tablet_manager.cc:1918] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:44.006242 13895 log.cc:1196] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/4c38277aaa3c4e6e8e098bec20a85f4d
I20260926 05:46:44.006456 13895 ts_tablet_manager.cc:1939] T 4c38277aaa3c4e6e8e098bec20a85f4d P ee1778851a0e4d5b851ec491cdee27bc: Deleting consensus metadata
I20260926 05:46:44.006860 13350 catalog_manager.cc:4841] TS ee1778851a0e4d5b851ec491cdee27bc (127.11.247.2:40657): tablet 4c38277aaa3c4e6e8e098bec20a85f4d (table test-table [id=1fc0b2fe52124472a74e67ee4a612ec6]) successfully deleted
Sep 26 05:46:44 dist-test-slave-21p4 krb5kdc[13319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401603, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:44.015189 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:44.006523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.11.247.1:35317)
0926 05:46:44.006572 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:44.006574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:44.007100 (+ 526us) server_negotiation.cc:407] Connection header received
0926 05:46:44.007177 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:44.007178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:44.007188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:44.007256 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:44.007816 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:44.008303 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:44.009045 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:44.009241 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:44.009324 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:44.012269 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:44.012279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:44.012280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:44.012288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:44.013922 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:44.014300 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:44.014303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:44.014304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:44.014366 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:44.014566 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:44.014567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:44.014568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:44.014755 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:44.014916 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:44.014963 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 05:46:44.015061 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 05:46:44.015630 13561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38504
I20260926 05:46:44.022933 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:44.016731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.11.247.62:33335)
0926 05:46:44.016777 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:44.016779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:44.016886 (+ 107us) server_negotiation.cc:407] Connection header received
0926 05:46:44.016965 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:44.016966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:44.016978 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:44.017033 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:44.017536 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:44.018050 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:44.018815 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:44.019021 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:44.019108 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:44.020313 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:44.020315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:44.020315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:44.020323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:44.021745 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:44.022100 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:44.022101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:44.022102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:44.022159 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:44.022352 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:44.022353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:44.022354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:44.022473 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:44.022581 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:44.022724 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:46:44.022806 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:46:44.023159 13363 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58888
I20260926 05:46:44.023743 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13436
I20260926 05:46:44.030095 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13576
I20260926 05:46:44.035503 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13716
I20260926 05:46:44.041265 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13332
2026-09-26T05:46:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4808 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:44 dist-test-slave-21p4 krb5kdc[13938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:44 dist-test-slave-21p4 krb5kdc[13938](info): set up 2 sockets
Sep 26 05:46:44 dist-test-slave-21p4 krb5kdc[13938](info): commencing operation
krb5kdc: starting...
W20260926 05:46:45.094434 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:45.226104 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39175
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:39175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:45.316385 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:45.316620 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:45.316644 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:45.317811 13951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:45.317854 13951 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:46:45.317868 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:45.317886 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:45.317899 13951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:45.317934 13951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:45.319621 13951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:39175
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:45.319875 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:45.320122 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:45.325074 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.325074 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.325122 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:45.325173 13951 server_base.cc:1034] running on GCE node
I20260926 05:46:45.325613 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:45.325913 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:45.327072 13951 hybrid_clock.cc:648] HybridClock initialized: now 1790401605327064 us; error 34 us; skew 500 ppm
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:45.329450 13951 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:45.329875 13951 webserver.cc:460] Webserver started at http://127.11.247.62:36231/ using document root <none> and password file <none>
I20260926 05:46:45.330127 13951 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:45.330193 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:45.330333 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:45.331525 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "d017f6631cb2409aa8a7f85a8de1f8d2"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "070faa48fb5de898acaf3e4f4bed3bd3"
server_key_iv: "0ad6f3f8f6b6cce9d063eeab2773f437"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.331894 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "d017f6631cb2409aa8a7f85a8de1f8d2"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "070faa48fb5de898acaf3e4f4bed3bd3"
server_key_iv: "0ad6f3f8f6b6cce9d063eeab2773f437"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.333237 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:46:45.333959 13966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.334121 13951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:45.334215 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "d017f6631cb2409aa8a7f85a8de1f8d2"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "070faa48fb5de898acaf3e4f4bed3bd3"
server_key_iv: "0ad6f3f8f6b6cce9d063eeab2773f437"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.334296 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:45.390260 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:45.392414 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:45.392584 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:45.397189 13951 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:39175
I20260926 05:46:45.397312 14028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:39175 every 8 connection(s)
I20260926 05:46:45.397599 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:45.398280 14029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:45.400640 14029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Bootstrap starting.
I20260926 05:46:45.401245 14029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:45.401551 14029 log.cc:824] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:45.401669 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 13951
I20260926 05:46:45.401772 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:45.401942 12252 external_mini_cluster.cc:1428] Setting key 2d258062d177c2b28685146561c711f9
I20260926 05:46:45.402330 14029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: No bootstrap required, opened a new log
I20260926 05:46:45.404690 14029 raft_consensus.cc:348] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39175 } }
I20260926 05:46:45.404868 14029 raft_consensus.cc:374] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:45.404934 14029 raft_consensus.cc:729] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d017f6631cb2409aa8a7f85a8de1f8d2, State: Initialized, Role: FOLLOWER
I20260926 05:46:45.405110 14029 consensus_queue.cc:260] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39175 } }
I20260926 05:46:45.405203 14029 raft_consensus.cc:388] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:45.405251 14029 raft_consensus.cc:482] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:45.405301 14029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:45.405900 14029 raft_consensus.cc:504] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39175 } }
I20260926 05:46:45.406018 14029 leader_election.cc:302] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d017f6631cb2409aa8a7f85a8de1f8d2; no voters:
I20260926 05:46:45.406208 14029 leader_election.cc:288] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:45.406267 14034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:45.406451 14034 raft_consensus.cc:686] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [term 1 LEADER]: Becoming Leader. State: Replica: d017f6631cb2409aa8a7f85a8de1f8d2, State: Running, Role: LEADER
I20260926 05:46:45.406507 14029 sys_catalog.cc:564] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:45.406693 14034 consensus_queue.cc:237] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39175 } }
I20260926 05:46:45.407138 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d017f6631cb2409aa8a7f85a8de1f8d2. Latest consensus state: current_term: 1 leader_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39175 } } }
I20260926 05:46:45.407150 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d017f6631cb2409aa8a7f85a8de1f8d2" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39175 } } }
I20260926 05:46:45.407264 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:45.407225 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:45.407895 14042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:45.408565 14042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:45.410435 14042 catalog_manager.cc:1347] Generated new cluster ID: 9d0825cb602247efa6e1228b2c4282e6
I20260926 05:46:45.410507 14042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:45.416093 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.402449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.11.247.62:39175)
0926 05:46:45.402708 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.402718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.402749 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:46:45.404700 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.404711 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.404779 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.404988 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.405808 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.407716 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.408861 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.409504 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.409623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.412447 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.412452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.412452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.412465 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.414414 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.414833 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.414836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.414837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.414966 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.415156 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.415157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.415157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.415481 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.415622 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.415722 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.415871 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":144,"threads_started":1}
I20260926 05:46:45.438843 14042 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:45.439603 14042 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:45.448194 14042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Generated new TSK 0
I20260926 05:46:45.448431 14042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:45.477602 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39175
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:45.565798 14055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:45.566066 14055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:45.566112 14055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:45.567423 14055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:45.567490 14055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:45.567526 14055 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:46:45.567556 14055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:46:45.567590 14055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:45.569116 14055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:45.569416 14055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:45.569659 14055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:46:45.575114 14055 server_base.cc:1034] running on GCE node
W20260926 05:46:45.575081 14063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.575076 14060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.575081 14061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:45.575560 14055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:45.575831 14055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:45.577003 14055 hybrid_clock.cc:648] HybridClock initialized: now 1790401605576994 us; error 38 us; skew 500 ppm
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:45.579663 14055 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:45.580129 14055 webserver.cc:460] Webserver started at http://127.11.247.1:37991/ using document root <none> and password file <none>
I20260926 05:46:45.580369 14055 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:45.580435 14055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:45.580586 14055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:45.581665 14055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "10ce54b0e7bf457c8b2203cd0d54c42c"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "5e28283b31811bf0a62537144ecc6cbf"
server_key_iv: "d516373ce6f67bfb1c3ed1dab4c24f65"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.582024 14055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "10ce54b0e7bf457c8b2203cd0d54c42c"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "5e28283b31811bf0a62537144ecc6cbf"
server_key_iv: "d516373ce6f67bfb1c3ed1dab4c24f65"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.583534 14055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:46:45.584316 14070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.584478 14055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:45.584578 14055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "10ce54b0e7bf457c8b2203cd0d54c42c"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "5e28283b31811bf0a62537144ecc6cbf"
server_key_iv: "d516373ce6f67bfb1c3ed1dab4c24f65"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.584649 14055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:45.599865 14055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:45.601790 14055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:45.601953 14055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:45.602492 14055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:45.602558 14055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.602669 14055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:45.602712 14055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:45.610664 14055 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:43803
I20260926 05:46:45.610745 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:43803 every 8 connection(s)
I20260926 05:46:45.611095 14055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:45.613497 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.603208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:35475 (local address 127.11.247.62:39175)
0926 05:46:45.603301 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.603304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.603553 (+ 249us) server_negotiation.cc:407] Connection header received
0926 05:46:45.604365 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.604366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.604385 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.604458 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.605130 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.605997 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.607244 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.607471 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.607563 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.610326 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.610328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.610328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.610338 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.612002 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.612423 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.612424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.612425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.612487 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.612744 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.612745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.612746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.612875 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.612951 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.613303 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.613422 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 05:46:45.613540 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14055
I20260926 05:46:45.613624 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:45.613705 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.603184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.11.247.1:35475)
0926 05:46:45.603395 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.603448 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.604201 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.604591 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.604596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.604847 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:45.605051 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.605056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.606116 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.606117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.607131 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.607135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.607593 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.607594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.607756 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.608461 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:45.608470 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:45.610211 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:45.612101 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.612108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.612111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.612344 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.612560 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.612561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.612561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.612671 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.612949 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:45.612958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:45.613100 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.613228 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.613444 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260926 05:46:45.613816 12252 external_mini_cluster.cc:1428] Setting key 740202111bab31da8c0f1d3e64e64695
I20260926 05:46:45.614516 14186 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39175
I20260926 05:46:45.614668 14186 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:45.614926 14186 heartbeater.cc:511] Master 127.11.247.62:39175 requested a full tablet report, sending...
I20260926 05:46:45.615552 13982 ts_manager.cc:194] Registered new tserver with Master: 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803)
I20260926 05:46:45.616490 13982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:35475
I20260926 05:46:45.621635 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.615387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.11.247.1:58871)
0926 05:46:45.615564 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.615576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.615662 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.615816 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.615816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.616021 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:45.616085 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.616087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.616993 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.616994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.617699 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.617702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.618076 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.618077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.618157 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.618681 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:45.618686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:45.618997 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:45.620585 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.620586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.620586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.620765 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.620969 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.620970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.620971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.621035 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.621275 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:45.621276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:45.621325 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.621407 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.621521 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260926 05:46:45.621626 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.615547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:58871 (local address 127.11.247.62:39175)
0926 05:46:45.615588 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.615590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.615635 (+ 45us) server_negotiation.cc:407] Connection header received
0926 05:46:45.615741 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.615742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.615753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.615812 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.616175 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.616853 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.617776 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.617970 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.618073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.619098 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.619099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.619100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.619106 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.620500 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.620847 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.620849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.620849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.620903 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.621101 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.621102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.621102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.621210 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.621284 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.621432 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.621520 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:45.641949 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39175
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:45.730865 14195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:45.731092 14195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:45.731120 14195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:45.732265 14195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:45.732299 14195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:45.732342 14195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:46:45.732354 14195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:46:45.732378 14195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:45.733850 14195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:45.734064 14195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:45.734256 14195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:45.739790 14203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:45.739938 14195 server_base.cc:1034] running on GCE node
W20260926 05:46:45.739774 14200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.739835 14201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:45.740229 14195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:45.740518 14195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:45.741707 14195 hybrid_clock.cc:648] HybridClock initialized: now 1790401605741687 us; error 40 us; skew 500 ppm
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:45.744462 14195 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:45.745023 14195 webserver.cc:460] Webserver started at http://127.11.247.2:34831/ using document root <none> and password file <none>
I20260926 05:46:45.745281 14195 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:45.745352 14195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:45.745507 14195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:45.746721 14195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "ae5445622d734d0f929e3dc078cdf03c"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "5dd821fc1d0417646088b1ad4557baa3"
server_key_iv: "95b857363a265a4df40faa323d808a39"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.747082 14195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5445622d734d0f929e3dc078cdf03c"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "5dd821fc1d0417646088b1ad4557baa3"
server_key_iv: "95b857363a265a4df40faa323d808a39"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.748528 14195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:46:45.749297 14210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.749481 14195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:45.749583 14195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "ae5445622d734d0f929e3dc078cdf03c"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "5dd821fc1d0417646088b1ad4557baa3"
server_key_iv: "95b857363a265a4df40faa323d808a39"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.749657 14195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:45.773967 14195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:45.776108 14195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:45.776275 14195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:45.776901 14195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:45.776950 14195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.776981 14195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:45.777005 14195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:45.784173 14195 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:42953
I20260926 05:46:45.784255 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:42953 every 8 connection(s)
I20260926 05:46:45.784709 14195 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:45.786386 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.777301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:39993 (local address 127.11.247.62:39175)
0926 05:46:45.777366 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.777368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.777700 (+ 332us) server_negotiation.cc:407] Connection header received
0926 05:46:45.778303 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.778304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.778320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.778415 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.779139 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.780038 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.781135 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.781333 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.781404 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.783654 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.783656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.783657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.783665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.785314 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.785671 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.785672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.785672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.785727 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.785911 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.785911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.785912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.786021 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.786069 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.786263 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.786323 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:46:45.786653 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.777301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.11.247.2:39993)
0926 05:46:45.777554 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.777596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.778163 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.778512 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.778518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.778839 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:45.779042 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.779047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.780139 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.780141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.781041 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.781044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.781447 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.781448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.781567 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.782063 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:45.782077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:45.783552 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:45.785371 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.785374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.785376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.785606 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.785780 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.785780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.785781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.785866 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.786059 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:45.786062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:45.786174 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.786251 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.786370 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
I20260926 05:46:45.787123 14326 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39175
I20260926 05:46:45.787212 14326 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:45.787403 14326 heartbeater.cc:511] Master 127.11.247.62:39175 requested a full tablet report, sending...
I20260926 05:46:45.787783 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14195
I20260926 05:46:45.787791 13983 ts_manager.cc:194] Registered new tserver with Master: ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953)
I20260926 05:46:45.787876 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:45.788079 12252 external_mini_cluster.cc:1428] Setting key 77f20bd6372e3d4e4aa29b876f7d9089
I20260926 05:46:45.788240 13983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:39993
I20260926 05:46:45.794770 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.788345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.11.247.2:36343)
0926 05:46:45.788623 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.788635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.788698 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.788857 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.788859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.789100 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:45.789178 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.789180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.789844 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.789845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.790564 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.790566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.791117 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.791118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.791176 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.791647 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:45.791651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:45.791924 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:45.793703 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.793704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.793704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.793866 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.794073 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.794074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.794074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.794123 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.794478 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:45.794479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:45.794517 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.794610 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.794700 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":90,"threads_started":1}
I20260926 05:46:45.794852 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.788310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:36343 (local address 127.11.247.62:39175)
0926 05:46:45.788351 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.788353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.788700 (+ 347us) server_negotiation.cc:407] Connection header received
0926 05:46:45.788749 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.788750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.788762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.788812 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.789279 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.789739 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.790688 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.791028 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.791119 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.792024 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.792025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.792025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.792033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.793621 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.793941 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.793942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.793943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.794008 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.794177 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.794178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.794178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.794399 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.794493 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.794635 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.794723 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:45.815650 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39175
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:45.901099 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:45.901310 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:45.901338 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:45.902522 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:45.902566 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:45.902642 14335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:46:45.902663 14335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:46:45.902681 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:45.904124 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:45.904317 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:45.904503 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:45.909554 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.909560 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:45.909569 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:45.909850 14335 server_base.cc:1034] running on GCE node
I20260926 05:46:45.910040 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:45.910238 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:45.911391 14335 hybrid_clock.cc:648] HybridClock initialized: now 1790401605911357 us; error 50 us; skew 500 ppm
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:45.913936 14335 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:45.914398 14335 webserver.cc:460] Webserver started at http://127.11.247.3:36445/ using document root <none> and password file <none>
I20260926 05:46:45.914669 14335 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:45.914718 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:45.914810 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:45.915858 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "370424a46afc439bba85327eb1225b03"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "283bdadee8bf6805ef803a2305938520"
server_key_iv: "32676218ba6b4b8282e7ca754f8c4238"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.916224 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "370424a46afc439bba85327eb1225b03"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "283bdadee8bf6805ef803a2305938520"
server_key_iv: "32676218ba6b4b8282e7ca754f8c4238"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.917567 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:45.918264 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.918432 14335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:45.918521 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "370424a46afc439bba85327eb1225b03"
format_stamp: "Formatted at 2026-09-26 05:46:45 on dist-test-slave-21p4"
server_key: "283bdadee8bf6805ef803a2305938520"
server_key_iv: "32676218ba6b4b8282e7ca754f8c4238"
server_key_version: "encryptionkey@0"
I20260926 05:46:45.918622 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:45.945968 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:45.948549 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:45.948709 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:45.949273 14335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:45.949327 14335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:45.949368 14335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:45.949390 14335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:45.956812 14335 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:40183
I20260926 05:46:45.956892 14465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:40183 every 8 connection(s)
I20260926 05:46:45.957235 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:45.959359 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.949680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:60413 (local address 127.11.247.62:39175)
0926 05:46:45.949732 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.949735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.950018 (+ 283us) server_negotiation.cc:407] Connection header received
0926 05:46:45.950783 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.950785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.950799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.950868 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.951835 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.952355 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.953642 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.953873 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.953955 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.956608 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.956610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.956610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.956619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.958166 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.958569 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.958570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.958570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.958663 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.958849 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.958849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.958850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.958953 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.959001 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.959235 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.959295 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:46:45.959520 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.949677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.11.247.3:60413)
0926 05:46:45.949843 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.949899 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.950663 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.950966 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.950972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.951391 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:45.951717 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.951725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.952468 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.952470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.953542 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.953545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.953985 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.953987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.954120 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.954695 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:45.954706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:45.956505 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:45.958239 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.958262 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.958264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.958500 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.958708 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.958709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.958709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.958799 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.959006 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:45.959011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:45.959127 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.959210 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.959328 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260926 05:46:45.959992 14466 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39175
I20260926 05:46:45.960090 14466 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:45.960350 14466 heartbeater.cc:511] Master 127.11.247.62:39175 requested a full tablet report, sending...
I20260926 05:46:45.960711 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14335
I20260926 05:46:45.960765 13983 ts_manager.cc:194] Registered new tserver with Master: 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:45.960798 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:45.960994 12252 external_mini_cluster.cc:1428] Setting key 0211f0f4c295422fc5aa10092fb9af0a
I20260926 05:46:45.961273 13983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:60413
I20260926 05:46:45.962405 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:46:45.967015 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.961250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:49871 (local address 127.11.247.62:39175)
0926 05:46:45.961290 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.961292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.961305 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:46:45.961407 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.961408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.961427 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.961475 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.961814 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.962376 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.963135 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.963436 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.963502 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.964485 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.964487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.964487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.964494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.965827 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.966226 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.966227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.966227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.966275 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.966481 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.966482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.966482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.966676 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.966730 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.966899 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.966953 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:46:45.967125 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.961112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.11.247.3:49871)
0926 05:46:45.961255 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.961267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.961321 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.961481 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.961482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.961676 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:45.961735 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.961738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.962473 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.962474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.963056 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.963058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.963518 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.963519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.963598 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.964031 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:45.964036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:45.964400 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:45.965906 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.965907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.965908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.966150 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.966334 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:45.966335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:45.966335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:45.966408 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:45.966740 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:45.966741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:45.966796 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.966884 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.967004 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:46:45 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:45.986210 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.979141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.11.247.62:39175)
0926 05:46:45.979205 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.979207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.979215 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:46:45.979241 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.979242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.979251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.979307 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:45.979904 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.980395 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.981252 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.981479 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.981542 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.983863 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:45.983868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:45.983869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:45.983876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:45.985083 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.985438 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.985440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.985440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.985488 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:45.985685 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:45.985686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:45.985686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:45.985792 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:45.985891 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.986021 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.986103 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:46:45.987968 13983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:45.989753 13983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:45.996762 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.992431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.11.247.2:42953)
0926 05:46:45.992753 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.992758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.992770 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:46:45.992812 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.992814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.992829 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.992911 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:45.993105 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.994267 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.995626 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.996363 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.996440 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.996499 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.996650 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.996684 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":97,"threads_started":1}
I20260926 05:46:45.996767 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.992306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:42953 (local address 127.0.0.1:60760)
0926 05:46:45.992632 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.992644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.992697 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.992936 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.992940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.992942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:45.993018 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.993020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.994346 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.994347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.995526 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.995528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.996459 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.996460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.996541 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.996557 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.996632 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.996685 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":56,"thread_start_us":88,"threads_started":1}
I20260926 05:46:45.997213 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.992275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:43803 (local address 127.0.0.1:48226)
0926 05:46:45.992479 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.992522 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.992590 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.993221 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.993222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.993223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:45.993274 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.993276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.994111 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.994112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.995505 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.995508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.996970 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.996972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.997044 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.997048 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.997098 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.997144 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260926 05:46:45.997479 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.992396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.11.247.1:43803)
0926 05:46:45.993047 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.993052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.993061 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:45.993099 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.993101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.993116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.993220 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:45.993344 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.994028 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.995584 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.996364 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.996441 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.996499 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.997322 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.997377 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":128,"threads_started":1}
I20260926 05:46:45.997493 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.992486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40183 (local address 127.0.0.1:38768)
0926 05:46:45.992633 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:45.992644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:45.992697 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:45.993109 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:45.993110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:45.993111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:45.993154 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.993155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.995508 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.995509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.996227 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:45.996229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.997268 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.997269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:45.997327 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.997331 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:45.997381 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:46:45.997417 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260926 05:46:45.997516 14260 tablet_service.cc:1467] Processing CreateTablet for tablet 76abbbe2e42b453f9df22b4c12005606 (DEFAULT_TABLE table=test-table [id=dbdad111abc94b3fa794e3c28e6d00f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:45.997742 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:45.992701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.11.247.3:40183)
0926 05:46:45.992862 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:45.992868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:45.992892 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:46:45.992936 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:45.992938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:45.992952 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:45.993040 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:45.993230 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.994262 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.996326 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:45.997121 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:45.997178 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:45.997232 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:45.997614 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 05:46:45.997655 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260926 05:46:45.997927 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76abbbe2e42b453f9df22b4c12005606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:45.999244 14400 tablet_service.cc:1467] Processing CreateTablet for tablet 76abbbe2e42b453f9df22b4c12005606 (DEFAULT_TABLE table=test-table [id=dbdad111abc94b3fa794e3c28e6d00f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:45.999281 14120 tablet_service.cc:1467] Processing CreateTablet for tablet 76abbbe2e42b453f9df22b4c12005606 (DEFAULT_TABLE table=test-table [id=dbdad111abc94b3fa794e3c28e6d00f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:45.999583 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76abbbe2e42b453f9df22b4c12005606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:45.999583 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76abbbe2e42b453f9df22b4c12005606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.001058 14489 tablet_bootstrap.cc:492] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Bootstrap starting.
I20260926 05:46:46.001725 14489 tablet_bootstrap.cc:654] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.002048 14489 log.cc:824] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:46.002815 14489 tablet_bootstrap.cc:492] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: No bootstrap required, opened a new log
I20260926 05:46:46.002890 14489 ts_tablet_manager.cc:1397] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:46:46.003055 14491 tablet_bootstrap.cc:492] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Bootstrap starting.
I20260926 05:46:46.004052 14491 tablet_bootstrap.cc:654] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.004381 14491 log.cc:824] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:46.005172 14491 tablet_bootstrap.cc:492] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: No bootstrap required, opened a new log
I20260926 05:46:46.005103 14489 raft_consensus.cc:348] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.005240 14491 ts_tablet_manager.cc:1397] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:46.005280 14489 raft_consensus.cc:374] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.005331 14489 raft_consensus.cc:729] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5445622d734d0f929e3dc078cdf03c, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.005455 14489 consensus_queue.cc:260] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.005743 14489 ts_tablet_manager.cc:1428] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:46:46.005771 14326 heartbeater.cc:503] Master 127.11.247.62:39175 was elected leader, sending a full tablet report...
I20260926 05:46:46.005949 14492 tablet_bootstrap.cc:492] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Bootstrap starting.
I20260926 05:46:46.006734 14492 tablet_bootstrap.cc:654] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.007066 14492 log.cc:824] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:46.007886 14492 tablet_bootstrap.cc:492] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: No bootstrap required, opened a new log
I20260926 05:46:46.007970 14492 ts_tablet_manager.cc:1397] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:46:46.007901 14491 raft_consensus.cc:348] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.008054 14491 raft_consensus.cc:374] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.008095 14491 raft_consensus.cc:729] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370424a46afc439bba85327eb1225b03, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.008208 14491 consensus_queue.cc:260] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.008462 14491 ts_tablet_manager.cc:1428] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:46:46.008466 14466 heartbeater.cc:503] Master 127.11.247.62:39175 was elected leader, sending a full tablet report...
I20260926 05:46:46.009936 14492 raft_consensus.cc:348] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.010067 14492 raft_consensus.cc:374] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.010088 14492 raft_consensus.cc:729] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.010191 14492 consensus_queue.cc:260] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.010401 14492 ts_tablet_manager.cc:1428] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:46:46.010468 14186 heartbeater.cc:503] Master 127.11.247.62:39175 was elected leader, sending a full tablet report...
W20260926 05:46:46.035363 14327 tablet.cc:2369] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:46:46.111977 14187 tablet.cc:2369] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:46:46.183806 14497 raft_consensus.cc:482] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:46.184003 14497 raft_consensus.cc:504] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.184516 14497 leader_election.cc:288] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183), ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953)
I20260926 05:46:46.187568 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.184741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:58747 (local address 127.11.247.3:40183)
0926 05:46:46.184806 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.184807 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.184822 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:46:46.184939 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.184941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.184952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.185037 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:46.185241 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.185747 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.186835 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.187208 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.187289 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.187298 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.187461 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.187501 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:46:46.187574 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.184634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40183 (local address 127.11.247.1:58747)
0926 05:46:46.184768 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.184779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.184837 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.185047 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.185048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.185051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:46.185152 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.185154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.185848 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.185850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.186719 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.186721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.187301 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.187302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.187372 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.187376 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.187452 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.187498 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260926 05:46:46.187686 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.184762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:42953 (local address 127.11.247.1:57911)
0926 05:46:46.184902 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.184912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.184986 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.185144 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.185145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.185146 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:46.185209 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.185211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.185925 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.185925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.186804 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.186807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.187374 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.187375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.187443 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.187447 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.187516 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.187555 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260926 05:46:46.187660 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.184830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:57911 (local address 127.11.247.2:42953)
0926 05:46:46.184873 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.184876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.185012 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:46:46.185048 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.185049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.185060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.185140 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:46.185301 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.185834 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.186875 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.187270 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.187374 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.187383 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.187521 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.187561 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:46:46.188030 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76abbbe2e42b453f9df22b4c12005606" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c" is_pre_election: true
I20260926 05:46:46.188036 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76abbbe2e42b453f9df22b4c12005606" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03" is_pre_election: true
I20260926 05:46:46.188187 14280 raft_consensus.cc:2454] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:46.188191 14420 raft_consensus.cc:2454] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:46.188403 14072 leader_election.cc:302] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, 370424a46afc439bba85327eb1225b03; no voters:
I20260926 05:46:46.188566 14497 raft_consensus.cc:2781] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:46.188622 14497 raft_consensus.cc:482] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:46.188653 14497 raft_consensus.cc:3037] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.189327 14497 raft_consensus.cc:504] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.189473 14497 leader_election.cc:288] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Requested vote from peers 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183), ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953)
I20260926 05:46:46.189610 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76abbbe2e42b453f9df22b4c12005606" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c"
I20260926 05:46:46.189698 14280 raft_consensus.cc:3037] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.189667 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76abbbe2e42b453f9df22b4c12005606" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03"
I20260926 05:46:46.189779 14420 raft_consensus.cc:3037] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.190403 14280 raft_consensus.cc:2454] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:46.190541 14420 raft_consensus.cc:2454] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:46.190647 14071 leader_election.cc:302] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:46.190752 14497 raft_consensus.cc:2781] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:46.190905 14497 raft_consensus.cc:686] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 LEADER]: Becoming Leader. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Running, Role: LEADER
I20260926 05:46:46.191018 14497 consensus_queue.cc:237] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } }
I20260926 05:46:46.191923 13983 catalog_manager.cc:5495] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1). New cstate: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } health_report { overall_health: UNKNOWN } } }
W20260926 05:46:46.207892 14467 tablet.cc:2369] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:46:46 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:46.254954 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.247427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39175 (local address 127.0.0.1:53106)
0926 05:46:46.247651 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.247669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.247731 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.247886 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.247886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.248206 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:46.248255 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.248258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.248987 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.248988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.249602 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.249604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.249966 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.249967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.250010 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.250515 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:46.250520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:46.252003 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:46.253655 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:46.253664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:46.253666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:46.253929 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:46.254107 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:46.254108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:46.254108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:46.254164 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:46.254527 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:46.254535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:46.254643 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.254723 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.254836 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":150,"threads_started":1}
I20260926 05:46:46.254997 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.247514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.11.247.62:39175)
0926 05:46:46.247585 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.247587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.247735 (+ 148us) server_negotiation.cc:407] Connection header received
0926 05:46:46.247780 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.247781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.247791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.247872 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:46.248376 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.248899 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.249697 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.249878 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.249940 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.252082 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:46.252083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:46.252084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:46.252090 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:46.253584 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:46.253996 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:46.253997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:46.253998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:46.254050 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:46.254247 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:46.254248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:46.254248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:46.254432 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:46.254502 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.254771 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.254885 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":35}
I20260926 05:46:46.257179 13983 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:53106:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:46:46.257417 13983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:46.259002 14400 tablet_service.cc:1467] Processing CreateTablet for tablet eb4298ed759e4084821fb9b9a10eca6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcd83460a2234f869902cd6e8e89f7b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:46:46.259006 14260 tablet_service.cc:1467] Processing CreateTablet for tablet eb4298ed759e4084821fb9b9a10eca6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcd83460a2234f869902cd6e8e89f7b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:46:46.259021 14120 tablet_service.cc:1467] Processing CreateTablet for tablet eb4298ed759e4084821fb9b9a10eca6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcd83460a2234f869902cd6e8e89f7b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:46:46.259176 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4298ed759e4084821fb9b9a10eca6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.259176 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4298ed759e4084821fb9b9a10eca6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.259198 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4298ed759e4084821fb9b9a10eca6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.260427 14492 tablet_bootstrap.cc:492] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: Bootstrap starting.
I20260926 05:46:46.260684 14489 tablet_bootstrap.cc:492] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: Bootstrap starting.
I20260926 05:46:46.260710 14491 tablet_bootstrap.cc:492] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: Bootstrap starting.
I20260926 05:46:46.260842 14492 tablet_bootstrap.cc:654] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.261243 14491 tablet_bootstrap.cc:654] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.261433 14489 tablet_bootstrap.cc:654] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.261968 14492 tablet_bootstrap.cc:492] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: No bootstrap required, opened a new log
I20260926 05:46:46.262023 14492 ts_tablet_manager.cc:1397] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:46:46.262084 14489 tablet_bootstrap.cc:492] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: No bootstrap required, opened a new log
I20260926 05:46:46.262110 14491 tablet_bootstrap.cc:492] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: No bootstrap required, opened a new log
I20260926 05:46:46.262127 14489 ts_tablet_manager.cc:1397] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:46.262148 14491 ts_tablet_manager.cc:1397] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:46.262220 14492 raft_consensus.cc:348] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.262279 14492 raft_consensus.cc:374] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.262292 14492 raft_consensus.cc:729] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.262287 14489 raft_consensus.cc:348] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.262287 14491 raft_consensus.cc:348] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.262352 14492 consensus_queue.cc:260] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.262373 14489 raft_consensus.cc:374] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.262377 14491 raft_consensus.cc:374] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.262398 14489 raft_consensus.cc:729] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5445622d734d0f929e3dc078cdf03c, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.262398 14491 raft_consensus.cc:729] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370424a46afc439bba85327eb1225b03, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.262461 14492 ts_tablet_manager.cc:1428] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:46.262441 14489 consensus_queue.cc:260] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.262442 14491 consensus_queue.cc:260] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.262540 14489 ts_tablet_manager.cc:1428] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:46.262549 14491 ts_tablet_manager.cc:1428] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:46.262522 14497 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.262626 14497 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.262696 14494 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.262701 14496 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.262768 14496 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.262768 14494 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.325824 14497 raft_consensus.cc:482] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:46.325922 14497 raft_consensus.cc:504] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.326097 14497 leader_election.cc:288] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953), 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:46.326304 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4298ed759e4084821fb9b9a10eca6b" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c" is_pre_election: true
I20260926 05:46:46.326404 14280 raft_consensus.cc:2454] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:46.326412 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4298ed759e4084821fb9b9a10eca6b" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03" is_pre_election: true
I20260926 05:46:46.326545 14420 raft_consensus.cc:2454] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:46.326552 14071 leader_election.cc:302] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:46.326668 14497 raft_consensus.cc:2781] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:46.326710 14497 raft_consensus.cc:482] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:46.326732 14497 raft_consensus.cc:3037] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.327415 14497 raft_consensus.cc:504] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.327574 14497 leader_election.cc:288] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Requested vote from peers ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953), 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:46.327759 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4298ed759e4084821fb9b9a10eca6b" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03"
I20260926 05:46:46.327728 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4298ed759e4084821fb9b9a10eca6b" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c"
I20260926 05:46:46.327829 14280 raft_consensus.cc:3037] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.327829 14420 raft_consensus.cc:3037] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.328570 14420 raft_consensus.cc:2454] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:46.328570 14280 raft_consensus.cc:2454] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:46.328788 14072 leader_election.cc:302] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, 370424a46afc439bba85327eb1225b03; no voters:
I20260926 05:46:46.328887 14497 raft_consensus.cc:2781] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:46.328961 14497 raft_consensus.cc:686] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 LEADER]: Becoming Leader. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Running, Role: LEADER
I20260926 05:46:46.329027 14497 consensus_queue.cc:237] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.329430 14500 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: TxnStatusTablet state changed. Reason: New leader 10ce54b0e7bf457c8b2203cd0d54c42c. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.329531 14500 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:46:46.329762 14512 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:46:46.329761 13982 catalog_manager.cc:5495] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1). New cstate: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } health_report { overall_health: UNKNOWN } } }
I20260926 05:46:46.329918 14512 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:46:46.338907 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.335992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.11.247.1:43803)
0926 05:46:46.336041 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.336043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.336119 (+ 76us) server_negotiation.cc:407] Connection header received
0926 05:46:46.336234 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.336235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.336257 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.336332 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:46.336493 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.337016 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.338112 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.338301 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.338354 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.338563 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.338699 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.338757 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.338836 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.338853 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:46:46.338965 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.335921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:43803 (local address 127.0.0.1:48240)
0926 05:46:46.336066 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.336079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.336143 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.336370 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.336371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.336373 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:46.336426 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.336429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.337090 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.337091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.338021 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.338024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.338372 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.338374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.338447 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.338466 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.338760 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.338769 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.338832 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.338886 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 05:46:46.340871 14280 raft_consensus.cc:1261] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:46.340874 14420 raft_consensus.cc:1261] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:46.341184 14500 consensus_queue.cc:1035] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:46.341346 14497 consensus_queue.cc:1035] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:46.342525 14494 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: TxnStatusTablet state changed. Reason: New leader 10ce54b0e7bf457c8b2203cd0d54c42c. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.342645 14494 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.342567 14496 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: TxnStatusTablet state changed. Reason: New leader 10ce54b0e7bf457c8b2203cd0d54c42c. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.342694 14496 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.343693 14500 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.343778 14500 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:46:46.344564 14494 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.344604 14516 mvcc.cc:205] Tried to move back new op lower bound from 7333484979569565696 to 7333484979524034560. Current Snapshot: MvccSnapshot[applied={T|T < 7333484979569565696 or (T in {7333484979569565696})}]
I20260926 05:46:46.344655 14494 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P ae5445622d734d0f929e3dc078cdf03c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.344712 14497 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.344779 14497 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 10ce54b0e7bf457c8b2203cd0d54c42c: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:46:46.345341 14496 tablet_replica.cc:439] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } } }
I20260926 05:46:46.345422 14496 tablet_replica.cc:442] T eb4298ed759e4084821fb9b9a10eca6b P 370424a46afc439bba85327eb1225b03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:46:46.348981 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.346660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.11.247.1:43803)
0926 05:46:46.346719 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.346721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.346729 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:46:46.346751 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.346752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.346762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.346834 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:46.346972 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.347471 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.348237 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.348459 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.348545 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.348704 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.348739 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.348797 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.348869 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.348891 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:46:46.354425 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.351466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:42483 (local address 127.11.247.1:43803)
0926 05:46:46.351506 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.351508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.351636 (+ 128us) server_negotiation.cc:407] Connection header received
0926 05:46:46.351679 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.351679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.351689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.351747 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:46.351887 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.352625 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.353506 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.353805 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.353878 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.354173 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.354212 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.354273 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.354331 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.354347 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20}
I20260926 05:46:46.355787 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.351413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:43803 (local address 127.11.247.1:42483)
0926 05:46:46.351565 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.351574 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.351631 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.351764 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.351765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.351766 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:46.351829 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.351830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.352693 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.352694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.353432 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.353434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.353904 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.353905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.353986 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.354000 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.354275 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.354277 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.354324 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.354375 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260926 05:46:46.358280 14280 raft_consensus.cc:1261] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:46.358276 14420 raft_consensus.cc:1261] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:46.358455 14500 consensus_queue.cc:1035] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:46.358547 14497 consensus_queue.cc:1035] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:46.359683 14515 mvcc.cc:205] Tried to move back new op lower bound from 7333484979641913344 to 7333484978959179776. Current Snapshot: MvccSnapshot[applied={T|T < 7333484979641913344}]
I20260926 05:46:46.372463 13982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:46.372577 13982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:46.373359 13982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Sending DeleteTablet for 3 replicas of tablet 76abbbe2e42b453f9df22b4c12005606
I20260926 05:46:46.373642 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet 76abbbe2e42b453f9df22b4c12005606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:46 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 05:46:46.373665 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet 76abbbe2e42b453f9df22b4c12005606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:46 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38768
I20260926 05:46:46.373772 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet 76abbbe2e42b453f9df22b4c12005606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:46 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60760
I20260926 05:46:46.374042 14533 tablet_replica.cc:330] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: stopping tablet replica
I20260926 05:46:46.374171 14533 raft_consensus.cc:2229] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:46.374233 14533 raft_consensus.cc:2258] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:46.374274 14531 tablet_replica.cc:330] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: stopping tablet replica
I20260926 05:46:46.374244 13982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:46.374380 13982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:46.374396 14531 raft_consensus.cc:2229] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:46.374449 14531 raft_consensus.cc:2258] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:46.374645 14532 tablet_replica.cc:330] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: stopping tablet replica
I20260926 05:46:46.374745 14532 raft_consensus.cc:2229] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:46:46.374851 14532 raft_consensus.cc:2258] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:46.375085 14531 ts_tablet_manager.cc:1905] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:46.375437 14533 ts_tablet_manager.cc:1905] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:46.375653 14260 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e (DEFAULT_TABLE table=test-table [id=86b0f5ec636b419a830ed32c6c53fd0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:46.375679 14400 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e (DEFAULT_TABLE table=test-table [id=86b0f5ec636b419a830ed32c6c53fd0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:46.375759 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.375775 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.377075 14533 ts_tablet_manager.cc:1918] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:46.377099 14491 tablet_bootstrap.cc:492] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Bootstrap starting.
I20260926 05:46:46.377133 14489 tablet_bootstrap.cc:492] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Bootstrap starting.
I20260926 05:46:46.377138 14533 log.cc:1196] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/76abbbe2e42b453f9df22b4c12005606
I20260926 05:46:46.377449 14533 ts_tablet_manager.cc:1939] T 76abbbe2e42b453f9df22b4c12005606 P ae5445622d734d0f929e3dc078cdf03c: Deleting consensus metadata
I20260926 05:46:46.377542 14489 tablet_bootstrap.cc:654] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.377545 14491 tablet_bootstrap.cc:654] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.377570 14531 ts_tablet_manager.cc:1918] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:46.377637 14531 log.cc:1196] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/76abbbe2e42b453f9df22b4c12005606
I20260926 05:46:46.377912 14531 ts_tablet_manager.cc:1939] T 76abbbe2e42b453f9df22b4c12005606 P 370424a46afc439bba85327eb1225b03: Deleting consensus metadata
I20260926 05:46:46.378090 13967 catalog_manager.cc:4841] TS ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953): tablet 76abbbe2e42b453f9df22b4c12005606 (table test-table [id=dbdad111abc94b3fa794e3c28e6d00f9]) successfully deleted
I20260926 05:46:46.378281 14119 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e (DEFAULT_TABLE table=test-table [id=86b0f5ec636b419a830ed32c6c53fd0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:46.378297 13968 catalog_manager.cc:4841] TS 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183): tablet 76abbbe2e42b453f9df22b4c12005606 (table test-table [id=dbdad111abc94b3fa794e3c28e6d00f9]) successfully deleted
I20260926 05:46:46.378404 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:46.378538 14491 tablet_bootstrap.cc:492] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: No bootstrap required, opened a new log
I20260926 05:46:46.378633 14491 ts_tablet_manager.cc:1397] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:46:46.378777 14491 raft_consensus.cc:348] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.378854 14491 raft_consensus.cc:374] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.378881 14491 raft_consensus.cc:729] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370424a46afc439bba85327eb1225b03, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.378937 14491 consensus_queue.cc:260] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.379031 14491 ts_tablet_manager.cc:1428] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:46.379818 14492 tablet_bootstrap.cc:492] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Bootstrap starting.
I20260926 05:46:46.380043 14532 ts_tablet_manager.cc:1905] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:46.380300 14492 tablet_bootstrap.cc:654] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:46.380873 14492 tablet_bootstrap.cc:492] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: No bootstrap required, opened a new log
I20260926 05:46:46.380918 14492 ts_tablet_manager.cc:1397] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:46.381054 14492 raft_consensus.cc:348] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.381150 14492 raft_consensus.cc:374] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.381177 14492 raft_consensus.cc:729] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.381222 14492 consensus_queue.cc:260] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.381327 14492 ts_tablet_manager.cc:1428] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:46.381543 14532 ts_tablet_manager.cc:1918] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:46.381618 14532 log.cc:1196] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/76abbbe2e42b453f9df22b4c12005606
I20260926 05:46:46.381876 14532 ts_tablet_manager.cc:1939] T 76abbbe2e42b453f9df22b4c12005606 P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting consensus metadata
I20260926 05:46:46.382308 14489 tablet_bootstrap.cc:492] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: No bootstrap required, opened a new log
I20260926 05:46:46.382371 14489 ts_tablet_manager.cc:1397] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 05:46:46.382527 14489 raft_consensus.cc:348] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.382642 14489 raft_consensus.cc:374] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:46.382665 14489 raft_consensus.cc:729] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5445622d734d0f929e3dc078cdf03c, State: Initialized, Role: FOLLOWER
I20260926 05:46:46.382654 13968 catalog_manager.cc:4841] TS 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803): tablet 76abbbe2e42b453f9df22b4c12005606 (table test-table [id=dbdad111abc94b3fa794e3c28e6d00f9]) successfully deleted
I20260926 05:46:46.382712 14489 consensus_queue.cc:260] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.382805 14489 ts_tablet_manager.cc:1428] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:46.442667 14496 raft_consensus.cc:482] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:46.442800 14496 raft_consensus.cc:504] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.443220 14496 leader_election.cc:288] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803), ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953)
I20260926 05:46:46.446786 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.443414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:43803 (local address 127.11.247.3:46193)
0926 05:46:46.443475 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.443488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.443533 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.443799 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.443800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.443802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:46.443880 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.443883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.444571 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.444573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.445757 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.445760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.446498 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.446498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.446573 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.446577 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.446641 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.446685 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 05:46:46.446786 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.443554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:46193 (local address 127.11.247.1:43803)
0926 05:46:46.443610 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.443613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.443623 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:46.443655 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.443656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.443667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.443741 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:46.443967 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.444478 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.445887 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.446404 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.446480 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.446492 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.446654 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.446709 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:46:46.447049 14140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ef1f9d9374fd7a8f53216c55fd36e" candidate_uuid: "370424a46afc439bba85327eb1225b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" is_pre_election: true
I20260926 05:46:46.447185 14140 raft_consensus.cc:2454] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 370424a46afc439bba85327eb1225b03 in term 0.
I20260926 05:46:46.447391 14352 leader_election.cc:302] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, 370424a46afc439bba85327eb1225b03; no voters:
I20260926 05:46:46.447479 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.443638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:42595 (local address 127.11.247.2:42953)
0926 05:46:46.443684 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.443687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.443706 (+ 19us) server_negotiation.cc:407] Connection header received
0926 05:46:46.443734 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.443735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.443748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.443805 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:46.444043 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.444779 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.446465 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.447090 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.447161 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.447169 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.447354 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.447405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:46:46.447538 14496 raft_consensus.cc:2781] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:46.447569 14496 raft_consensus.cc:482] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:46.447588 14496 raft_consensus.cc:3037] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.447734 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.443414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:42953 (local address 127.11.247.3:42595)
0926 05:46:46.443613 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.443623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.443665 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.443811 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.443812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.443814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:46.443903 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.443904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.444893 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.444894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.446373 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.446375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.447202 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.447203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.447278 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.447283 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.447588 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.447645 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":33,"thread_start_us":81,"threads_started":1}
I20260926 05:46:46.448010 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ef1f9d9374fd7a8f53216c55fd36e" candidate_uuid: "370424a46afc439bba85327eb1225b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c" is_pre_election: true
I20260926 05:46:46.448115 14280 raft_consensus.cc:2454] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 370424a46afc439bba85327eb1225b03 in term 0.
I20260926 05:46:46.448513 14496 raft_consensus.cc:504] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.448665 14496 leader_election.cc:288] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [CANDIDATE]: Term 1 election: Requested vote from peers 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803), ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953)
I20260926 05:46:46.448854 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ef1f9d9374fd7a8f53216c55fd36e" candidate_uuid: "370424a46afc439bba85327eb1225b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c"
I20260926 05:46:46.448944 14280 raft_consensus.cc:3037] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.449574 14280 raft_consensus.cc:2454] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370424a46afc439bba85327eb1225b03 in term 1.
I20260926 05:46:46.449790 14351 leader_election.cc:302] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370424a46afc439bba85327eb1225b03, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:46.449824 14140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4ef1f9d9374fd7a8f53216c55fd36e" candidate_uuid: "370424a46afc439bba85327eb1225b03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c"
I20260926 05:46:46.449890 14496 raft_consensus.cc:2781] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:46.449922 14140 raft_consensus.cc:3037] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:46.450017 14496 raft_consensus.cc:686] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 1 LEADER]: Becoming Leader. State: Replica: 370424a46afc439bba85327eb1225b03, State: Running, Role: LEADER
I20260926 05:46:46.450090 14496 consensus_queue.cc:237] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:46.450802 14140 raft_consensus.cc:2454] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370424a46afc439bba85327eb1225b03 in term 1.
I20260926 05:46:46.450858 13980 catalog_manager.cc:5495] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 370424a46afc439bba85327eb1225b03 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "370424a46afc439bba85327eb1225b03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } health_report { overall_health: HEALTHY } } }
I20260926 05:46:46.476154 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.473411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.11.247.3:40183)
0926 05:46:46.473452 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.473455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.473521 (+ 66us) server_negotiation.cc:407] Connection header received
0926 05:46:46.473606 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.473606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.473624 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.473681 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:46.473826 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.474319 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.475407 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.475606 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.475659 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.475820 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.475923 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.475975 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.476042 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.476060 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:46:46.481139 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.478624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:43803 (local address 127.11.247.3:59887)
0926 05:46:46.478798 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.478808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.478868 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.478997 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.478998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.478999 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:46.479057 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.479058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.479709 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.479710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.480300 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.480303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.480680 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.480681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.480753 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.480767 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.480950 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.480952 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.481013 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.481062 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260926 05:46:46.481119 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.478672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:59887 (local address 127.11.247.1:43803)
0926 05:46:46.478718 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.478721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.478852 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:46:46.478923 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.478924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.478934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.478988 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:46.479136 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.479629 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.480387 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.480597 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.480678 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.480855 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.480894 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.480950 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.481018 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.481037 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:46:46.483421 14140 raft_consensus.cc:1261] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Refusing update from remote peer 370424a46afc439bba85327eb1225b03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:46.483704 14280 raft_consensus.cc:1261] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Refusing update from remote peer 370424a46afc439bba85327eb1225b03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:46.483940 14496 consensus_queue.cc:1035] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:46.484083 14538 consensus_queue.cc:1035] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:46.485204 14517 mvcc.cc:205] Tried to move back new op lower bound from 7333484980154118144 to 7333484980020305920. Current Snapshot: MvccSnapshot[applied={T|T < 7333484980154118144}]
I20260926 05:46:46.492975 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.490180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:38199 (local address 127.11.247.3:40183)
0926 05:46:46.490219 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:46.490221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:46.490230 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:46.490294 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:46.490294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:46.490306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:46.490367 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:46.490482 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.491256 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.492269 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.492567 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.492641 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.492741 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.492775 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.492824 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:46.492870 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 05:46:46.492891 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:46:46.492960 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:46.490094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40183 (local address 127.11.247.1:38199)
0926 05:46:46.490187 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:46.490199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:46.490239 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:46.490368 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:46.490369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:46.490370 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:46:46.490420 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.490422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.491346 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.491346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.492190 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:46.492193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:46.492637 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:46.492638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:46.492677 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:46.492683 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:46:46.492831 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:46:46.492832 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:46:46.492867 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 05:46:46.492903 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 05:46:46.493412 14517 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 05:46:48.019161 13980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:48.019330 13980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:48.020474 13980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Sending DeleteTablet for 3 replicas of tablet 6d4ef1f9d9374fd7a8f53216c55fd36e
I20260926 05:46:48.020803 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:48 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38768
I20260926 05:46:48.020817 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:48 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 05:46:48.021080 14568 tablet_replica.cc:330] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: stopping tablet replica
I20260926 05:46:48.021068 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4ef1f9d9374fd7a8f53216c55fd36e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:48 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60760
I20260926 05:46:48.021173 14568 raft_consensus.cc:2229] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:48.021248 14568 raft_consensus.cc:2258] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:48.021466 14567 tablet_replica.cc:330] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: stopping tablet replica
I20260926 05:46:48.021584 14568 ts_tablet_manager.cc:1905] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:48.021592 14567 raft_consensus.cc:2229] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:46:48.021751 14567 raft_consensus.cc:2258] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:48.022102 14567 ts_tablet_manager.cc:1905] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:48.022915 14569 tablet_replica.cc:330] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: stopping tablet replica
I20260926 05:46:48.023001 14569 raft_consensus.cc:2229] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:48.023096 14569 raft_consensus.cc:2258] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:48.023370 14569 ts_tablet_manager.cc:1905] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 05:46:48 dist-test-slave-21p4 krb5kdc[13938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 05:46:48.057051 14568 ts_tablet_manager.cc:1918] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:46:48.057051 14567 ts_tablet_manager.cc:1918] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:46:48.057142 14567 log.cc:1196] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/6d4ef1f9d9374fd7a8f53216c55fd36e
I20260926 05:46:48.057142 14568 log.cc:1196] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/6d4ef1f9d9374fd7a8f53216c55fd36e
I20260926 05:46:48.057199 14569 ts_tablet_manager.cc:1918] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:46:48.057268 14569 log.cc:1196] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/6d4ef1f9d9374fd7a8f53216c55fd36e
I20260926 05:46:48.057376 14568 ts_tablet_manager.cc:1939] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting consensus metadata
I20260926 05:46:48.057456 14567 ts_tablet_manager.cc:1939] T 6d4ef1f9d9374fd7a8f53216c55fd36e P 370424a46afc439bba85327eb1225b03: Deleting consensus metadata
I20260926 05:46:48.057503 14569 ts_tablet_manager.cc:1939] T 6d4ef1f9d9374fd7a8f53216c55fd36e P ae5445622d734d0f929e3dc078cdf03c: Deleting consensus metadata
Sep 26 05:46:48 dist-test-slave-21p4 krb5kdc[13938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:48.072335 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:48.058877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.11.247.62:39175)
0926 05:46:48.059074 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:48.059076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:48.059165 (+ 89us) server_negotiation.cc:407] Connection header received
0926 05:46:48.059235 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:48.059236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:48.059248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:48.059319 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:48.059936 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:48.060548 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:48.061434 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:48.061636 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:48.061693 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:48.069238 (+ 7545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:48.069241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:48.069241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:48.069251 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:48.071150 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:48.071553 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:48.071554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:48.071554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:48.071614 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:48.071837 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:48.071838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:48.071838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:48.071969 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:48.072049 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:48.072183 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 05:46:48.072257 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260926 05:46:48.074826 13980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:48.075186 13980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:48.075426 13968 catalog_manager.cc:4841] TS 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803): tablet 6d4ef1f9d9374fd7a8f53216c55fd36e (table test-table [id=86b0f5ec636b419a830ed32c6c53fd0d]) successfully deleted
I20260926 05:46:48.075485 13967 catalog_manager.cc:4841] TS ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953): tablet 6d4ef1f9d9374fd7a8f53216c55fd36e (table test-table [id=86b0f5ec636b419a830ed32c6c53fd0d]) successfully deleted
I20260926 05:46:48.075505 13968 catalog_manager.cc:4841] TS 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183): tablet 6d4ef1f9d9374fd7a8f53216c55fd36e (table test-table [id=86b0f5ec636b419a830ed32c6c53fd0d]) successfully deleted
I20260926 05:46:48.077128 14400 tablet_service.cc:1467] Processing CreateTablet for tablet b4b876c91b71446783e29c3f71f5e30c (DEFAULT_TABLE table=test-table [id=641a5d22286646ada1c152f3f69f5cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:48.077113 14119 tablet_service.cc:1467] Processing CreateTablet for tablet b4b876c91b71446783e29c3f71f5e30c (DEFAULT_TABLE table=test-table [id=641a5d22286646ada1c152f3f69f5cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:48.077152 14260 tablet_service.cc:1467] Processing CreateTablet for tablet b4b876c91b71446783e29c3f71f5e30c (DEFAULT_TABLE table=test-table [id=641a5d22286646ada1c152f3f69f5cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:48.077267 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b876c91b71446783e29c3f71f5e30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:48.077270 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b876c91b71446783e29c3f71f5e30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:48.077301 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b876c91b71446783e29c3f71f5e30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:48.078778 14578 tablet_bootstrap.cc:492] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Bootstrap starting.
I20260926 05:46:48.078979 14580 tablet_bootstrap.cc:492] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Bootstrap starting.
I20260926 05:46:48.079157 14578 tablet_bootstrap.cc:654] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:48.079427 14580 tablet_bootstrap.cc:654] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:48.079758 14578 tablet_bootstrap.cc:492] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: No bootstrap required, opened a new log
I20260926 05:46:48.079788 14579 tablet_bootstrap.cc:492] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Bootstrap starting.
I20260926 05:46:48.079869 14578 ts_tablet_manager.cc:1397] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:48.080018 14578 raft_consensus.cc:348] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.080134 14578 raft_consensus.cc:374] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:48.080166 14578 raft_consensus.cc:729] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5445622d734d0f929e3dc078cdf03c, State: Initialized, Role: FOLLOWER
I20260926 05:46:48.080222 14579 tablet_bootstrap.cc:654] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:48.080224 14578 consensus_queue.cc:260] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.080428 14578 ts_tablet_manager.cc:1428] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:48.080875 14580 tablet_bootstrap.cc:492] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: No bootstrap required, opened a new log
I20260926 05:46:48.080945 14580 ts_tablet_manager.cc:1397] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:46:48.081068 14580 raft_consensus.cc:348] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.081172 14580 raft_consensus.cc:374] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:48.081203 14580 raft_consensus.cc:729] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Initialized, Role: FOLLOWER
I20260926 05:46:48.081439 14579 tablet_bootstrap.cc:492] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: No bootstrap required, opened a new log
I20260926 05:46:48.081498 14579 ts_tablet_manager.cc:1397] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:46:48.081614 14579 raft_consensus.cc:348] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.081704 14579 raft_consensus.cc:374] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:48.081727 14579 raft_consensus.cc:729] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370424a46afc439bba85327eb1225b03, State: Initialized, Role: FOLLOWER
I20260926 05:46:48.081790 14579 consensus_queue.cc:260] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.081913 14579 ts_tablet_manager.cc:1428] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:48.081979 14580 consensus_queue.cc:260] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.082126 14580 ts_tablet_manager.cc:1428] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:46:48.152695 14562 raft_consensus.cc:482] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:48.152823 14562 raft_consensus.cc:504] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.153025 14562 leader_election.cc:288] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953), 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:48.153312 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b876c91b71446783e29c3f71f5e30c" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c" is_pre_election: true
I20260926 05:46:48.153416 14280 raft_consensus.cc:2454] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:48.153324 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b876c91b71446783e29c3f71f5e30c" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03" is_pre_election: true
I20260926 05:46:48.153517 14420 raft_consensus.cc:2454] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:48.153630 14071 leader_election.cc:302] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:48.153734 14562 raft_consensus.cc:2781] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:48.153791 14562 raft_consensus.cc:482] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:48.153807 14562 raft_consensus.cc:3037] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:48.199548 14562 raft_consensus.cc:504] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.199805 14562 leader_election.cc:288] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Requested vote from peers ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953), 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:48.200014 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b876c91b71446783e29c3f71f5e30c" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c"
I20260926 05:46:48.200116 14280 raft_consensus.cc:3037] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:48.200089 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b876c91b71446783e29c3f71f5e30c" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03"
I20260926 05:46:48.200178 14420 raft_consensus.cc:3037] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:48.200987 14420 raft_consensus.cc:2454] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:48.200989 14280 raft_consensus.cc:2454] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:48.201205 14072 leader_election.cc:302] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, 370424a46afc439bba85327eb1225b03, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:48.201339 14562 raft_consensus.cc:2781] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:48.201381 14562 raft_consensus.cc:686] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 LEADER]: Becoming Leader. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Running, Role: LEADER
I20260926 05:46:48.201447 14562 consensus_queue.cc:237] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.202354 13982 catalog_manager.cc:5495] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1). New cstate: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } health_report { overall_health: UNKNOWN } } }
I20260926 05:46:48.232547 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:48.229614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.11.247.1:43803)
0926 05:46:48.229785 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:48.229787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:48.229793 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:46:48.229813 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:48.229813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:48.229823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:48.229901 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:48.230086 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:48.230725 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:48.231538 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:48.231764 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:48.231840 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:48.232170 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:48.232215 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:48.232336 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:48.232420 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:46:48.232446 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260926 05:46:48.235333 14280 raft_consensus.cc:1261] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:48.235349 14420 raft_consensus.cc:1261] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:48.235605 14497 consensus_queue.cc:1035] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:48.235792 14548 consensus_queue.cc:1035] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:48.247262 13982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:48.247385 13982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:48.248193 13982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Sending DeleteTablet for 3 replicas of tablet b4b876c91b71446783e29c3f71f5e30c
I20260926 05:46:48.248556 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b876c91b71446783e29c3f71f5e30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:48 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60760
I20260926 05:46:48.248554 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b876c91b71446783e29c3f71f5e30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:48 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38768
I20260926 05:46:48.248557 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b876c91b71446783e29c3f71f5e30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:48 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 05:46:48.248672 14567 tablet_replica.cc:330] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: stopping tablet replica
I20260926 05:46:48.248672 14569 tablet_replica.cc:330] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: stopping tablet replica
I20260926 05:46:48.248729 14567 raft_consensus.cc:2229] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:48.248730 14569 raft_consensus.cc:2229] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:48.248769 14567 raft_consensus.cc:2258] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:48.248773 14569 raft_consensus.cc:2258] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:48.248858 14568 tablet_replica.cc:330] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: stopping tablet replica
I20260926 05:46:48.248957 14568 raft_consensus.cc:2229] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:46:48.249034 14568 raft_consensus.cc:2258] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:48.249181 14569 ts_tablet_manager.cc:1905] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:48.249341 14567 ts_tablet_manager.cc:1905] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:48.250233 13982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:48.250418 13982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:48.250418 14568 ts_tablet_manager.cc:1905] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:48.252108 14400 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1ae5ead7d741bb87ef89eb32f52a41 (DEFAULT_TABLE table=test-table [id=b39a191fff6f4d09a3dc3a5b2d516d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:48.252076 14260 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1ae5ead7d741bb87ef89eb32f52a41 (DEFAULT_TABLE table=test-table [id=b39a191fff6f4d09a3dc3a5b2d516d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:48.252337 14119 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1ae5ead7d741bb87ef89eb32f52a41 (DEFAULT_TABLE table=test-table [id=b39a191fff6f4d09a3dc3a5b2d516d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:48.295624 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1ae5ead7d741bb87ef89eb32f52a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:48.295624 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1ae5ead7d741bb87ef89eb32f52a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:48.296499 14569 ts_tablet_manager.cc:1918] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:48.296564 14569 log.cc:1196] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/b4b876c91b71446783e29c3f71f5e30c
I20260926 05:46:48.296818 14569 ts_tablet_manager.cc:1939] T b4b876c91b71446783e29c3f71f5e30c P ae5445622d734d0f929e3dc078cdf03c: Deleting consensus metadata
I20260926 05:46:48.297140 14579 tablet_bootstrap.cc:492] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Bootstrap starting.
I20260926 05:46:48.297281 14578 tablet_bootstrap.cc:492] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Bootstrap starting.
I20260926 05:46:48.297504 14579 tablet_bootstrap.cc:654] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:48.297714 14578 tablet_bootstrap.cc:654] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:48.298177 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1ae5ead7d741bb87ef89eb32f52a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:48.298396 14579 tablet_bootstrap.cc:492] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: No bootstrap required, opened a new log
I20260926 05:46:48.298439 14579 ts_tablet_manager.cc:1397] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:48.298525 14578 tablet_bootstrap.cc:492] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: No bootstrap required, opened a new log
I20260926 05:46:48.298573 14578 ts_tablet_manager.cc:1397] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:48.298550 14579 raft_consensus.cc:348] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.298643 14579 raft_consensus.cc:374] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:48.298666 14579 raft_consensus.cc:729] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370424a46afc439bba85327eb1225b03, State: Initialized, Role: FOLLOWER
I20260926 05:46:48.298707 14579 consensus_queue.cc:260] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.298791 14579 ts_tablet_manager.cc:1428] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:48.298755 14578 raft_consensus.cc:348] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.298836 14578 raft_consensus.cc:374] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:48.298861 14578 raft_consensus.cc:729] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5445622d734d0f929e3dc078cdf03c, State: Initialized, Role: FOLLOWER
I20260926 05:46:48.298913 14578 consensus_queue.cc:260] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.299024 14578 ts_tablet_manager.cc:1428] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:48.299762 14580 tablet_bootstrap.cc:492] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c: Bootstrap starting.
I20260926 05:46:48.300168 14580 tablet_bootstrap.cc:654] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:48.300850 14580 tablet_bootstrap.cc:492] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c: No bootstrap required, opened a new log
I20260926 05:46:48.300978 14580 ts_tablet_manager.cc:1397] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:48.301174 14580 raft_consensus.cc:348] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.301273 14580 raft_consensus.cc:374] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:48.301291 14580 raft_consensus.cc:729] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Initialized, Role: FOLLOWER
I20260926 05:46:48.301350 14580 consensus_queue.cc:260] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.301539 14580 ts_tablet_manager.cc:1428] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:48.304855 14568 ts_tablet_manager.cc:1918] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:48.304984 14568 log.cc:1196] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/b4b876c91b71446783e29c3f71f5e30c
I20260926 05:46:48.305205 14568 ts_tablet_manager.cc:1939] T b4b876c91b71446783e29c3f71f5e30c P 10ce54b0e7bf457c8b2203cd0d54c42c: Deleting consensus metadata
I20260926 05:46:48.305796 14567 ts_tablet_manager.cc:1918] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:46:48.305948 14567 log.cc:1196] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/b4b876c91b71446783e29c3f71f5e30c
I20260926 05:46:48.306195 14567 ts_tablet_manager.cc:1939] T b4b876c91b71446783e29c3f71f5e30c P 370424a46afc439bba85327eb1225b03: Deleting consensus metadata
I20260926 05:46:48.311546 13967 catalog_manager.cc:4841] TS ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953): tablet b4b876c91b71446783e29c3f71f5e30c (table test-table [id=641a5d22286646ada1c152f3f69f5cd1]) successfully deleted
I20260926 05:46:48.347110 13968 catalog_manager.cc:4841] TS 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183): tablet b4b876c91b71446783e29c3f71f5e30c (table test-table [id=641a5d22286646ada1c152f3f69f5cd1]) successfully deleted
I20260926 05:46:48.378839 13968 catalog_manager.cc:4841] TS 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803): tablet b4b876c91b71446783e29c3f71f5e30c (table test-table [id=641a5d22286646ada1c152f3f69f5cd1]) successfully deleted
I20260926 05:46:48.535544 14497 raft_consensus.cc:482] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:48.535674 14497 raft_consensus.cc:504] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.535897 14497 leader_election.cc:288] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953), 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:48.536223 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1ae5ead7d741bb87ef89eb32f52a41" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03" is_pre_election: true
I20260926 05:46:48.536229 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1ae5ead7d741bb87ef89eb32f52a41" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c" is_pre_election: true
I20260926 05:46:48.536356 14280 raft_consensus.cc:2454] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:48.536381 14420 raft_consensus.cc:2454] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 0.
I20260926 05:46:48.536530 14071 leader_election.cc:302] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:48.536662 14497 raft_consensus.cc:2781] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:48.536705 14497 raft_consensus.cc:482] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:48.536732 14497 raft_consensus.cc:3037] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:48.564960 14497 raft_consensus.cc:504] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.565254 14497 leader_election.cc:288] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Requested vote from peers ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953), 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183)
I20260926 05:46:48.565502 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1ae5ead7d741bb87ef89eb32f52a41" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5445622d734d0f929e3dc078cdf03c"
I20260926 05:46:48.565606 14280 raft_consensus.cc:3037] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:48.565512 14420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1ae5ead7d741bb87ef89eb32f52a41" candidate_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370424a46afc439bba85327eb1225b03"
I20260926 05:46:48.565650 14420 raft_consensus.cc:3037] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:48.566354 14280 raft_consensus.cc:2454] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:48.566608 14420 raft_consensus.cc:2454] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ce54b0e7bf457c8b2203cd0d54c42c in term 1.
I20260926 05:46:48.566640 14071 leader_election.cc:302] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ce54b0e7bf457c8b2203cd0d54c42c, ae5445622d734d0f929e3dc078cdf03c; no voters:
I20260926 05:46:48.566764 14497 raft_consensus.cc:2781] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:48.566814 14497 raft_consensus.cc:686] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [term 1 LEADER]: Becoming Leader. State: Replica: 10ce54b0e7bf457c8b2203cd0d54c42c, State: Running, Role: LEADER
I20260926 05:46:48.566864 14497 consensus_queue.cc:237] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } }
I20260926 05:46:48.567559 13980 catalog_manager.cc:5495] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1). New cstate: current_term: 1 leader_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ce54b0e7bf457c8b2203cd0d54c42c" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 43803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 } health_report { overall_health: UNKNOWN } } }
I20260926 05:46:48.615447 14420 raft_consensus.cc:1261] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:48.615449 14280 raft_consensus.cc:1261] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Refusing update from remote peer 10ce54b0e7bf457c8b2203cd0d54c42c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:48.615705 14548 consensus_queue.cc:1035] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "370424a46afc439bba85327eb1225b03" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:48.615877 14497 consensus_queue.cc:1035] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 10ce54b0e7bf457c8b2203cd0d54c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5445622d734d0f929e3dc078cdf03c" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:46:48.620806 14546 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 05:46:50.140429 13982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:50.140585 13982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:50.141495 13982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d017f6631cb2409aa8a7f85a8de1f8d2: Sending DeleteTablet for 3 replicas of tablet 4d1ae5ead7d741bb87ef89eb32f52a41
I20260926 05:46:50.141800 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1ae5ead7d741bb87ef89eb32f52a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:50 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 05:46:50.141794 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1ae5ead7d741bb87ef89eb32f52a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:50 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60760
I20260926 05:46:50.141922 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14055
I20260926 05:46:50.142009 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1ae5ead7d741bb87ef89eb32f52a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:50 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38768
I20260926 05:46:50.142735 14616 tablet_replica.cc:330] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: stopping tablet replica
I20260926 05:46:50.142809 14618 tablet_replica.cc:330] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: stopping tablet replica
I20260926 05:46:50.142817 14616 raft_consensus.cc:2229] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:50.142877 14618 raft_consensus.cc:2229] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:50.142933 14618 raft_consensus.cc:2258] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:50.142948 14616 raft_consensus.cc:2258] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:50.143222 14616 ts_tablet_manager.cc:1905] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:50.143222 14618 ts_tablet_manager.cc:1905] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:50.145023 14618 ts_tablet_manager.cc:1918] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:46:50.145017 14616 ts_tablet_manager.cc:1918] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:46:50.145085 14618 log.cc:1196] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/4d1ae5ead7d741bb87ef89eb32f52a41
I20260926 05:46:50.145085 14616 log.cc:1196] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/4d1ae5ead7d741bb87ef89eb32f52a41
I20260926 05:46:50.145318 14616 ts_tablet_manager.cc:1939] T 4d1ae5ead7d741bb87ef89eb32f52a41 P ae5445622d734d0f929e3dc078cdf03c: Deleting consensus metadata
I20260926 05:46:50.145318 14618 ts_tablet_manager.cc:1939] T 4d1ae5ead7d741bb87ef89eb32f52a41 P 370424a46afc439bba85327eb1225b03: Deleting consensus metadata
I20260926 05:46:50.145668 13968 catalog_manager.cc:4841] TS 370424a46afc439bba85327eb1225b03 (127.11.247.3:40183): tablet 4d1ae5ead7d741bb87ef89eb32f52a41 (table test-table [id=b39a191fff6f4d09a3dc3a5b2d516d9a]) successfully deleted
I20260926 05:46:50.145733 13967 catalog_manager.cc:4841] TS ae5445622d734d0f929e3dc078cdf03c (127.11.247.2:42953): tablet 4d1ae5ead7d741bb87ef89eb32f52a41 (table test-table [id=b39a191fff6f4d09a3dc3a5b2d516d9a]) successfully deleted
W20260926 05:46:50.149912 13968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:43803 (error 108)
I20260926 05:46:50.150154 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14195
I20260926 05:46:50.151482 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:50.151228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:43803 (local address 127.0.0.1:48276)
0926 05:46:50.151389 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:50.151441 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260926 05:46:50.151659 13968 catalog_manager.cc:4584] TS 10ce54b0e7bf457c8b2203cd0d54c42c (127.11.247.1:43803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d1ae5ead7d741bb87ef89eb32f52a41: Network error: Client connection negotiation failed: client connection to 127.11.247.1:43803: connect: Connection refused (error 111)
I20260926 05:46:50.156054 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14335
I20260926 05:46:50.161886 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 13951
2026-09-26T05:46:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6121 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:50 dist-test-slave-21p4 krb5kdc[14623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:50 dist-test-slave-21p4 krb5kdc[14623](info): set up 2 sockets
Sep 26 05:46:50 dist-test-slave-21p4 krb5kdc[14623](info): commencing operation
krb5kdc: starting...
W20260926 05:46:51.210901 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:51.338171 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39629
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:39629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:51.422114 14636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:51.422399 14636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:51.422436 14636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:51.423741 14636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:51.423831 14636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:51.423854 14636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:51.423877 14636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:51.423906 14636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:51.425534 14636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:39629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:51.425863 14636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:51.426113 14636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:51.431735 14641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.431711 14644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.431701 14642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:51.431758 14636 server_base.cc:1034] running on GCE node
I20260926 05:46:51.432129 14636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:51.432502 14636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:51.433672 14636 hybrid_clock.cc:648] HybridClock initialized: now 1790401611433647 us; error 38 us; skew 500 ppm
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:51.436290 14636 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:51.436764 14636 webserver.cc:460] Webserver started at http://127.11.247.62:45537/ using document root <none> and password file <none>
I20260926 05:46:51.437013 14636 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:51.437083 14636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:51.437235 14636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:51.438361 14636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "2988d498c81244f7ae5fd728cab78417"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "ce5dae91011fe7ac61c07e1e99dffa47"
server_key_iv: "565a46e8043102901a991a7658698666"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.438797 14636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "2988d498c81244f7ae5fd728cab78417"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "ce5dae91011fe7ac61c07e1e99dffa47"
server_key_iv: "565a46e8043102901a991a7658698666"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.440277 14636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:51.441164 14651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.441429 14636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:46:51.441553 14636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "2988d498c81244f7ae5fd728cab78417"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "ce5dae91011fe7ac61c07e1e99dffa47"
server_key_iv: "565a46e8043102901a991a7658698666"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.441642 14636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:51.452186 14636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:51.454411 14636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:51.454648 14636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:51.458753 14636 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:39629
I20260926 05:46:51.458806 14703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:39629 every 8 connection(s)
I20260926 05:46:51.459190 14636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:51.460079 14704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:51.462387 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417: Bootstrap starting.
I20260926 05:46:51.463007 14704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:51.463253 14704 log.cc:824] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:51.463557 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14636
I20260926 05:46:51.463668 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:51.463879 12252 external_mini_cluster.cc:1428] Setting key e47784bb2b35cd864bea5434b3f5d06d
I20260926 05:46:51.464109 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417: No bootstrap required, opened a new log
I20260926 05:46:51.466303 14704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2988d498c81244f7ae5fd728cab78417" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39629 } }
I20260926 05:46:51.466507 14704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:51.466547 14704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2988d498c81244f7ae5fd728cab78417, State: Initialized, Role: FOLLOWER
I20260926 05:46:51.466729 14704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2988d498c81244f7ae5fd728cab78417" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39629 } }
I20260926 05:46:51.466828 14704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:51.466876 14704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:51.466925 14704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:51.467660 14704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2988d498c81244f7ae5fd728cab78417" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39629 } }
I20260926 05:46:51.467772 14704 leader_election.cc:302] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2988d498c81244f7ae5fd728cab78417; no voters:
I20260926 05:46:51.467931 14704 leader_election.cc:288] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:51.468011 14709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:51.468161 14704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:51.468160 14709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [term 1 LEADER]: Becoming Leader. State: Replica: 2988d498c81244f7ae5fd728cab78417, State: Running, Role: LEADER
I20260926 05:46:51.468384 14709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2988d498c81244f7ae5fd728cab78417" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39629 } }
I20260926 05:46:51.468724 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2988d498c81244f7ae5fd728cab78417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2988d498c81244f7ae5fd728cab78417" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39629 } } }
I20260926 05:46:51.468710 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2988d498c81244f7ae5fd728cab78417. Latest consensus state: current_term: 1 leader_uuid: "2988d498c81244f7ae5fd728cab78417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2988d498c81244f7ae5fd728cab78417" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39629 } } }
I20260926 05:46:51.468909 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:51.468916 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:51.469170 14718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:51.469913 14718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:51.471860 14718 catalog_manager.cc:1347] Generated new cluster ID: 7d76a7efaf784cf98c4160a0ad0ab8ef
I20260926 05:46:51.471911 14718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:51.477453 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:51.464421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.11.247.62:39629)
0926 05:46:51.464624 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:51.464636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:51.464673 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:46:51.465527 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:51.465538 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:51.465606 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:51.465764 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:51.466396 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.468164 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.469459 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.470131 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.470274 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:51.473243 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:51.473248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:51.473248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:51.473262 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:51.475464 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:51.475886 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:51.475891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:51.475891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:51.476018 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:51.476222 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:51.476223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:51.476223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:51.476596 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:51.476761 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:51.476874 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:46:51.477153 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260926 05:46:51.488236 14718 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:51.488796 14718 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:51.494143 14718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2988d498c81244f7ae5fd728cab78417: Generated new TSK 0
I20260926 05:46:51.494433 14718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:51.527305 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39629
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:51.613878 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:51.614101 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:51.614127 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:51.615289 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:51.615339 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:51.615367 14730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:51.616780 14730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:51.616981 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:51.617184 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:51.622315 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.622295 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.622298 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:51.622792 14730 server_base.cc:1034] running on GCE node
I20260926 05:46:51.623044 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:51.623313 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:51.624464 14730 hybrid_clock.cc:648] HybridClock initialized: now 1790401611624432 us; error 48 us; skew 500 ppm
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:51.627251 14730 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:51.627839 14730 webserver.cc:460] Webserver started at http://127.11.247.1:32919/ using document root <none> and password file <none>
I20260926 05:46:51.628080 14730 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:51.628126 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:51.628284 14730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:51.629446 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "4dd1374ff687489881a013496aaf9c6a"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "db0b2965c93d4843d9f6386834f4cdbd"
server_key_iv: "a87bcd0afd7ad883274f9997cc1c5805"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.629813 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "4dd1374ff687489881a013496aaf9c6a"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "db0b2965c93d4843d9f6386834f4cdbd"
server_key_iv: "a87bcd0afd7ad883274f9997cc1c5805"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.631474 14730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:51.632371 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.632706 14730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:51.632794 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "4dd1374ff687489881a013496aaf9c6a"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "db0b2965c93d4843d9f6386834f4cdbd"
server_key_iv: "a87bcd0afd7ad883274f9997cc1c5805"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.632892 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:51.657824 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:51.660341 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:51.660494 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:51.660760 14730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:51.661091 14730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:51.661162 14730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.661201 14730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:51.661228 14730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.667289 14730 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:41227
I20260926 05:46:51.667333 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:41227 every 8 connection(s)
I20260926 05:46:51.667671 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:51.673697 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14730
I20260926 05:46:51.673833 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:51.674027 12252 external_mini_cluster.cc:1428] Setting key f121034fe3176269f3dc12421edee797
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:51.678686 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:51.668393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:46909 (local address 127.11.247.62:39629)
0926 05:46:51.668440 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:51.668442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:51.668683 (+ 241us) server_negotiation.cc:407] Connection header received
0926 05:46:51.669432 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:51.669433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:51.669447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:51.669501 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:51.670576 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.671140 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.672214 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.672432 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.672521 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:51.675041 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:51.675042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:51.675043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:51.675052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:51.676847 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:51.677419 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:51.677420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:51.677421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:51.677478 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:51.677777 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:51.677779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:51.677779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:51.677941 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:51.678012 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:51.678415 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 05:46:51.678522 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:46:51.678814 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:51.668377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39629 (local address 127.11.247.1:46909)
0926 05:46:51.668523 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:51.668585 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:51.669317 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:51.669573 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:51.669577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:51.670273 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:51.670478 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:51.670483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.671256 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.671258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:51.672080 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:51.672082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.672535 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.672537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:51.672679 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:51.673247 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:51.673256 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:51.674829 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:51.676951 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:51.676961 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:51.676964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:51.677329 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:51.677550 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:51.677551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:51.677551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:51.677701 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:51.678076 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:51.678083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:51.678223 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 05:46:51.678369 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 05:46:51.678519 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260926 05:46:51.679378 14859 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39629
I20260926 05:46:51.679500 14859 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:51.679713 14859 heartbeater.cc:511] Master 127.11.247.62:39629 requested a full tablet report, sending...
I20260926 05:46:51.680404 14668 ts_manager.cc:194] Registered new tserver with Master: 4dd1374ff687489881a013496aaf9c6a (127.11.247.1:41227)
I20260926 05:46:51.681068 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:46909
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:51.702296 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39629
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:51.786712 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:51.786993 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:51.787040 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:51.788358 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:51.788444 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:51.788493 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:51.790055 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:51.790349 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:51.790650 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:51.796146 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.796126 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.796126 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:51.796255 14864 server_base.cc:1034] running on GCE node
I20260926 05:46:51.796684 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:51.796950 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:51.798133 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790401611798114 us; error 36 us; skew 500 ppm
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:51.800904 14864 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:51.801457 14864 webserver.cc:460] Webserver started at http://127.11.247.2:34785/ using document root <none> and password file <none>
I20260926 05:46:51.801705 14864 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:51.801777 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:51.801928 14864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:51.803126 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "83c02259fc9c49308662b2f4c1a48f4a"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "7ed7fb34d929e214cda7a6c3cfd02906"
server_key_iv: "6ca91dc2a2ff511443adfb9bc5483bca"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.803494 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "83c02259fc9c49308662b2f4c1a48f4a"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "7ed7fb34d929e214cda7a6c3cfd02906"
server_key_iv: "6ca91dc2a2ff511443adfb9bc5483bca"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.804908 14864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:51.805744 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.806001 14864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:51.806114 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "83c02259fc9c49308662b2f4c1a48f4a"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "7ed7fb34d929e214cda7a6c3cfd02906"
server_key_iv: "6ca91dc2a2ff511443adfb9bc5483bca"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.806202 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:51.832436 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:51.835105 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:51.835286 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:51.835484 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:51.835846 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:51.835927 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.835975 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:51.836018 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.842250 14864 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:34741
I20260926 05:46:51.842300 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:34741 every 8 connection(s)
I20260926 05:46:51.842667 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:51.847930 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14864
I20260926 05:46:51.848060 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:51.848318 12252 external_mini_cluster.cc:1428] Setting key 54fdd11ef303c83ee78d8ce9e5fa032c
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:51.856437 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:51.843434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:57323 (local address 127.11.247.62:39629)
0926 05:46:51.843524 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:51.843527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:51.843728 (+ 201us) server_negotiation.cc:407] Connection header received
0926 05:46:51.844553 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:51.844554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:51.844568 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:51.844627 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:51.845733 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.846268 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.847936 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.848339 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.848433 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:51.852738 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:51.852740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:51.852740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:51.852750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:51.854407 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:51.854985 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:51.854987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:51.854987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:51.855050 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:51.855348 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:51.855349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:51.855349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:51.855663 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:51.855739 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:51.856177 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 05:46:51.856280 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:46:51.857026 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:51.843405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39629 (local address 127.11.247.2:57323)
0926 05:46:51.843587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:51.843637 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:51.844438 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:51.844708 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:51.844714 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:51.845369 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:51.845623 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:51.845629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.846371 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.846372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:51.847766 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:51.847770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:51.848461 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:51.848462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:51.848649 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:51.850738 (+ 2089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:51.850752 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:51.852591 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:51.854518 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:51.854530 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:51.854533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:51.854887 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:51.855120 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:51.855121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:51.855121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:51.855277 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:51.855783 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:51.855792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:51.855994 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 05:46:51.856552 (+ 558us) client_negotiation.cc:240] Negotiation successful
0926 05:46:51.856767 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260926 05:46:51.858824 14993 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39629
I20260926 05:46:51.859035 14993 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:46:51.865505 14993 heartbeater.cc:511] Master 127.11.247.62:39629 requested a full tablet report, sending...
I20260926 05:46:51.866168 14668 ts_manager.cc:194] Registered new tserver with Master: 83c02259fc9c49308662b2f4c1a48f4a (127.11.247.2:34741)
I20260926 05:46:51.866680 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:57323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:51.887008 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39629
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:51.974076 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:51.974354 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:51.974397 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:51.975750 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:51.975831 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:51.975880 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:51.977481 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:51.977789 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:51.978041 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:51.983507 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:51.983616 14998 server_base.cc:1034] running on GCE node
W20260926 05:46:51.983498 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:51.983496 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:51.984110 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:51.984407 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:51.985596 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790401611985567 us; error 42 us; skew 500 ppm
Sep 26 05:46:51 dist-test-slave-21p4 krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:51.988502 14998 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:51.989051 14998 webserver.cc:460] Webserver started at http://127.11.247.3:35781/ using document root <none> and password file <none>
I20260926 05:46:51.989311 14998 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:51.989383 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:51.989544 14998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:51.990731 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "0c008cf29fe54d44961c9e766b2c2c25"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "7f0b98fa2b7d58e37bb8c34b622eb168"
server_key_iv: "26bcdbd903cc90e9529f677dc2e93bca"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.991173 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "0c008cf29fe54d44961c9e766b2c2c25"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "7f0b98fa2b7d58e37bb8c34b622eb168"
server_key_iv: "26bcdbd903cc90e9529f677dc2e93bca"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.992640 14998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:51.993410 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:51.993582 14998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:51.993691 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "0c008cf29fe54d44961c9e766b2c2c25"
format_stamp: "Formatted at 2026-09-26 05:46:51 on dist-test-slave-21p4"
server_key: "7f0b98fa2b7d58e37bb8c34b622eb168"
server_key_iv: "26bcdbd903cc90e9529f677dc2e93bca"
server_key_version: "encryptionkey@0"
I20260926 05:46:51.993793 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:52.012072 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:52.014714 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:52.014896 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:52.015154 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:52.015492 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:52.015551 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:52.015594 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:52.015638 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:52.021775 14998 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:35455
I20260926 05:46:52.021842 15126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:35455 every 8 connection(s)
I20260926 05:46:52.022162 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:52.022630 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 14998
I20260926 05:46:52.022744 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:52.022948 12252 external_mini_cluster.cc:1428] Setting key 5521b2d0015772c95192e96148049b42
Sep 26 05:46:52 dist-test-slave-21p4 krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:52.031947 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:52.022961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:33619 (local address 127.11.247.62:39629)
0926 05:46:52.023042 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:52.023044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:52.023291 (+ 247us) server_negotiation.cc:407] Connection header received
0926 05:46:52.024016 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:52.024017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:52.024029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:52.024085 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:52.025055 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:52.025572 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:52.026623 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:52.026848 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:52.026938 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:52.029135 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:52.029136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:52.029136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:52.029143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:52.030485 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:52.030952 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:52.030952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:52.030953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:52.031006 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:52.031296 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:52.031297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:52.031297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:52.031412 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:52.031471 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:52.031806 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 05:46:52.031875 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:46:52.032131 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:52.022945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39629 (local address 127.11.247.3:33619)
0926 05:46:52.023151 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:52.023216 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:52.023915 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:52.024159 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:52.024163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:52.024755 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:52.024955 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:52.024960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:52.025668 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:52.025669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:52.026509 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:52.026511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:52.026950 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:52.026951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:52.027091 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:52.027618 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:52.027627 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:52.029048 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:52.030562 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:52.030568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:52.030570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:52.030884 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:52.031105 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:52.031106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:52.031106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:52.031228 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:52.031514 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:52.031519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:52.031652 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 05:46:52.031777 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 05:46:52.031909 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
I20260926 05:46:52.032693 15127 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39629
I20260926 05:46:52.032797 15127 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:52.032949 15127 heartbeater.cc:511] Master 127.11.247.62:39629 requested a full tablet report, sending...
I20260926 05:46:52.033377 14668 ts_manager.cc:194] Registered new tserver with Master: 0c008cf29fe54d44961c9e766b2c2c25 (127.11.247.3:35455)
I20260926 05:46:52.033736 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:33619
I20260926 05:46:52.034870 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:46:52.046106 14708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:52.045100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.11.247.62:39629)
0926 05:46:52.045202 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:52.045204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:52.045272 (+ 68us) server_negotiation.cc:407] Connection header received
0926 05:46:52.045325 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:52.045326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:52.045344 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:52.045409 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:52.046015 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35680: BlockingRecv error: recv got EOF from 127.0.0.1:35680 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:46:52.047112 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14730
I20260926 05:46:52.052034 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14864
I20260926 05:46:52.056783 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14998
I20260926 05:46:52.061424 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 14636
2026-09-26T05:46:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1897 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:52 dist-test-slave-21p4 krb5kdc[15139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:52 dist-test-slave-21p4 krb5kdc[15139](info): set up 2 sockets
Sep 26 05:46:52 dist-test-slave-21p4 krb5kdc[15139](info): commencing operation
krb5kdc: starting...
W20260926 05:46:53.106688 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:53 dist-test-slave-21p4 krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:53.235042 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37475
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:53.321332 15152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:53.321607 15152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:53.321661 15152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:53.322971 15152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:53.323041 15152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:53.323064 15152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:53.323096 15152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:53.323141 15152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:53.324970 15152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:53.325284 15152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:53.325546 15152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:53.330739 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:53.330734 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:53.330734 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:53.331162 15152 server_base.cc:1034] running on GCE node
I20260926 05:46:53.331398 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:53.331697 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:53.332986 15152 hybrid_clock.cc:648] HybridClock initialized: now 1790401613332960 us; error 34 us; skew 500 ppm
Sep 26 05:46:53 dist-test-slave-21p4 krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401613, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:53.341149 15152 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:53.341674 15152 webserver.cc:460] Webserver started at http://127.11.247.62:46205/ using document root <none> and password file <none>
I20260926 05:46:53.341964 15152 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:53.342022 15152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:53.342141 15152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:53.343425 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "4b447bf4582548329f7c75cfe58032f7"
format_stamp: "Formatted at 2026-09-26 05:46:53 on dist-test-slave-21p4"
server_key: "dbe119b6d86fb80ddd13c784d87f8850"
server_key_iv: "d84b7399f1dec358ef32b64ab042f616"
server_key_version: "encryptionkey@0"
I20260926 05:46:53.343808 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "4b447bf4582548329f7c75cfe58032f7"
format_stamp: "Formatted at 2026-09-26 05:46:53 on dist-test-slave-21p4"
server_key: "dbe119b6d86fb80ddd13c784d87f8850"
server_key_iv: "d84b7399f1dec358ef32b64ab042f616"
server_key_version: "encryptionkey@0"
I20260926 05:46:53.345270 15152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:53.346122 15167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:53.346544 15152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:53.346722 15152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "4b447bf4582548329f7c75cfe58032f7"
format_stamp: "Formatted at 2026-09-26 05:46:53 on dist-test-slave-21p4"
server_key: "dbe119b6d86fb80ddd13c784d87f8850"
server_key_iv: "d84b7399f1dec358ef32b64ab042f616"
server_key_version: "encryptionkey@0"
I20260926 05:46:53.346809 15152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:53.367022 15152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:53.369333 15152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:53.369500 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:53.373679 15152 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:37475
I20260926 05:46:53.373747 15219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:37475 every 8 connection(s)
I20260926 05:46:53.374331 15152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:53.375031 15220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:53.377425 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7: Bootstrap starting.
I20260926 05:46:53.378075 15220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:53.378386 15220 log.cc:824] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:53.379199 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7: No bootstrap required, opened a new log
I20260926 05:46:53.380890 15220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b447bf4582548329f7c75cfe58032f7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37475 } }
I20260926 05:46:53.381052 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15152
I20260926 05:46:53.381067 15220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:53.381110 15220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b447bf4582548329f7c75cfe58032f7, State: Initialized, Role: FOLLOWER
I20260926 05:46:53.381155 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:53.381337 12252 external_mini_cluster.cc:1428] Setting key f1cb339cf2459227f739edaef255a27a
I20260926 05:46:53.381306 15220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b447bf4582548329f7c75cfe58032f7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37475 } }
I20260926 05:46:53.381412 15220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:53.381462 15220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:53.381515 15220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:53.382915 15220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b447bf4582548329f7c75cfe58032f7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37475 } }
I20260926 05:46:53.383050 15220 leader_election.cc:302] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b447bf4582548329f7c75cfe58032f7; no voters:
I20260926 05:46:53.383237 15220 leader_election.cc:288] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:53.383347 15225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:53.383515 15225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4b447bf4582548329f7c75cfe58032f7, State: Running, Role: LEADER
I20260926 05:46:53.383538 15220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:53.384996 15225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b447bf4582548329f7c75cfe58032f7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37475 } }
I20260926 05:46:53.385732 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b447bf4582548329f7c75cfe58032f7. Latest consensus state: current_term: 1 leader_uuid: "4b447bf4582548329f7c75cfe58032f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b447bf4582548329f7c75cfe58032f7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37475 } } }
I20260926 05:46:53.385842 15227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b447bf4582548329f7c75cfe58032f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b447bf4582548329f7c75cfe58032f7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37475 } } }
I20260926 05:46:53.386071 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:53.386082 15227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7 [sys.catalog]: This master's current role is: LEADER
W20260926 05:46:53.386888 15239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:53.386958 15239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:46:53.387036 15240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:53.387269 15240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:53.388876 15240 catalog_manager.cc:1347] Generated new cluster ID: c2915a2c5fd943a8a9e615f6de5b53d2
I20260926 05:46:53.388929 15240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:46:53 dist-test-slave-21p4 krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:53.393862 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:53.381875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.11.247.62:37475)
0926 05:46:53.382079 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:53.382090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:53.382126 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:46:53.382237 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:53.382243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:53.382297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:53.382421 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:53.383350 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:53.384933 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:53.386161 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:53.386855 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:53.387302 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:53.389893 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:53.389898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:53.389898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:53.389911 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:53.392004 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:53.392431 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:53.392435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:53.392435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:53.392570 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:53.392789 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:53.392790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:53.392791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:53.393220 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:53.393393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:53.393468 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 05:46:53.393615 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 05:46:53.394899 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:46:53.420147 15240 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:53.420936 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:53.416508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.11.247.62:37475)
0926 05:46:53.416557 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:53.416560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:53.416675 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:46:53.416734 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:53.416735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:53.416760 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:53.416847 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:53.417299 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:53.418085 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:53.419214 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:53.419549 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:53.419620 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:53.420445 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:53.420446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:53.420447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:46:53.420649 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:53.420666 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:46:53.420672 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:53.420747 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:53.420816 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 05:46:53.420842 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:46:53.420959 15240 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:53.432994 15240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b447bf4582548329f7c75cfe58032f7: Generated new TSK 0
I20260926 05:46:53.433259 15240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:46:53.438061 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15152
2026-09-26T05:46:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1375 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:53 dist-test-slave-21p4 krb5kdc[15253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:53 dist-test-slave-21p4 krb5kdc[15253](info): set up 2 sockets
Sep 26 05:46:53 dist-test-slave-21p4 krb5kdc[15253](info): commencing operation
krb5kdc: starting...
W20260926 05:46:54.496667 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 05:46:54.637763 12252 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.141s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 05:46:54.797614 12252 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.160s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:54 dist-test-slave-21p4 krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:54.882014 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33717
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:54.966964 15266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:54.967200 15266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:54.967226 15266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:54.968432 15266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:54.968473 15266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:54.968487 15266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:54.968500 15266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:54.968513 15266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:54.969996 15266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:54.970233 15266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:54.970432 15266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:54.975440 15271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:54.975449 15274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:54.975493 15272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:54.975745 15266 server_base.cc:1034] running on GCE node
I20260926 05:46:54.976017 15266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:54.976331 15266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:54.977504 15266 hybrid_clock.cc:648] HybridClock initialized: now 1790401614977496 us; error 40 us; skew 500 ppm
Sep 26 05:46:54 dist-test-slave-21p4 krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401614, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:54.980165 15266 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:54.980670 15266 webserver.cc:460] Webserver started at http://127.11.247.62:40255/ using document root <none> and password file <none>
I20260926 05:46:54.980937 15266 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:54.981012 15266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:54.981170 15266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:54.982311 15266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "74693284b17a438f88bc900b32458e96"
format_stamp: "Formatted at 2026-09-26 05:46:54 on dist-test-slave-21p4"
server_key: "9606d201fb75c4fcbbca21cb8b434f31"
server_key_iv: "3dcb8b530ecb806f6c4ec833fa8729d6"
server_key_version: "encryptionkey@0"
I20260926 05:46:54.982748 15266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "74693284b17a438f88bc900b32458e96"
format_stamp: "Formatted at 2026-09-26 05:46:54 on dist-test-slave-21p4"
server_key: "9606d201fb75c4fcbbca21cb8b434f31"
server_key_iv: "3dcb8b530ecb806f6c4ec833fa8729d6"
server_key_version: "encryptionkey@0"
I20260926 05:46:54.984125 15266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:54.984895 15281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:54.985069 15266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:54.985172 15266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "74693284b17a438f88bc900b32458e96"
format_stamp: "Formatted at 2026-09-26 05:46:54 on dist-test-slave-21p4"
server_key: "9606d201fb75c4fcbbca21cb8b434f31"
server_key_iv: "3dcb8b530ecb806f6c4ec833fa8729d6"
server_key_version: "encryptionkey@0"
I20260926 05:46:54.985285 15266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:54.996206 15266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:54.998500 15266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:54.998701 15266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:55.002456 15266 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33717
I20260926 05:46:55.002527 15333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33717 every 8 connection(s)
I20260926 05:46:55.002951 15266 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:55.003976 15334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:55.006409 15334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96: Bootstrap starting.
I20260926 05:46:55.007098 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15266
I20260926 05:46:55.007133 15334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:55.007215 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:55.007411 12252 external_mini_cluster.cc:1428] Setting key bc2cf82bd15feed691e00be1a169651b
I20260926 05:46:55.007450 15334 log.cc:824] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:55.008322 15334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96: No bootstrap required, opened a new log
I20260926 05:46:55.010838 15334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74693284b17a438f88bc900b32458e96" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33717 } }
I20260926 05:46:55.011047 15334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:55.011083 15334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74693284b17a438f88bc900b32458e96, State: Initialized, Role: FOLLOWER
I20260926 05:46:55.011212 15334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74693284b17a438f88bc900b32458e96" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33717 } }
I20260926 05:46:55.011308 15334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:55.011340 15334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:55.011390 15334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:55.012092 15334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74693284b17a438f88bc900b32458e96" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33717 } }
I20260926 05:46:55.012181 15334 leader_election.cc:302] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74693284b17a438f88bc900b32458e96; no voters:
I20260926 05:46:55.012421 15334 leader_election.cc:288] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:55.012465 15339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:55.012645 15339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [term 1 LEADER]: Becoming Leader. State: Replica: 74693284b17a438f88bc900b32458e96, State: Running, Role: LEADER
I20260926 05:46:55.012738 15334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:55.012821 15339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74693284b17a438f88bc900b32458e96" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33717 } }
I20260926 05:46:55.013260 15339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74693284b17a438f88bc900b32458e96. Latest consensus state: current_term: 1 leader_uuid: "74693284b17a438f88bc900b32458e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74693284b17a438f88bc900b32458e96" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33717 } } }
I20260926 05:46:55.013411 15339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:55.013387 15341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74693284b17a438f88bc900b32458e96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74693284b17a438f88bc900b32458e96" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33717 } } }
I20260926 05:46:55.013594 15341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:55.013710 15348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:55.014286 15348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:55.016902 15348 catalog_manager.cc:1347] Generated new cluster ID: 97b2ee709fd14b78824a14bd67288478
I20260926 05:46:55.016964 15348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:46:55.019935 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.007980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.11.247.62:33717)
0926 05:46:55.008158 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:55.008167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:55.008206 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:46:55.008705 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:55.008713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:55.008766 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:55.008890 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:55.009723 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.010794 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.011848 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.012442 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.012570 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.015884 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:55.015889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:55.015889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:55.015902 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:55.018154 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.018611 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.018614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.018615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.018724 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.018955 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.018956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.018957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.019283 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:55.019444 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:55.019558 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 05:46:55.019710 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260926 05:46:55.024004 15348 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:55.024633 15348 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:55.029664 15348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74693284b17a438f88bc900b32458e96: Generated new TSK 0
I20260926 05:46:55.029850 15348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:55.062124 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33717
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:55.146258 15360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:55.146522 15360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:55.146565 15360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:55.147907 15360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:55.147979 15360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:55.148027 15360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:55.149598 15360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:55.149895 15360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:55.150152 15360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:55.155848 15365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:55.155869 15366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:55.155905 15360 server_base.cc:1034] running on GCE node
W20260926 05:46:55.155869 15368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:55.156229 15360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:55.156461 15360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:55.157656 15360 hybrid_clock.cc:648] HybridClock initialized: now 1790401615157640 us; error 29 us; skew 500 ppm
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:55.160337 15360 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:55.160820 15360 webserver.cc:460] Webserver started at http://127.11.247.1:46163/ using document root <none> and password file <none>
I20260926 05:46:55.161095 15360 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:55.161201 15360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:55.161365 15360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:55.162496 15360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "b75c0a73fd2342e3850030f8cff3c833"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "4dfecd455d1b2ff8cd8b84a12b4599c5"
server_key_iv: "5d6cf06bab253a349c6267cf50d6eb4c"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.162964 15360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "b75c0a73fd2342e3850030f8cff3c833"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "4dfecd455d1b2ff8cd8b84a12b4599c5"
server_key_iv: "5d6cf06bab253a349c6267cf50d6eb4c"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.164495 15360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:55.165544 15375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.165781 15360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:55.165898 15360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "b75c0a73fd2342e3850030f8cff3c833"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "4dfecd455d1b2ff8cd8b84a12b4599c5"
server_key_iv: "5d6cf06bab253a349c6267cf50d6eb4c"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.165982 15360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:55.187657 15360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:55.190085 15360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:55.190249 15360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:55.190496 15360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:55.190864 15360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:55.190922 15360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.190963 15360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:55.191008 15360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.197141 15360 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:35741
I20260926 05:46:55.197191 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:35741 every 8 connection(s)
I20260926 05:46:55.197515 15360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:55.198105 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15360
I20260926 05:46:55.198213 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:55.198438 12252 external_mini_cluster.cc:1428] Setting key 67d4e76f773105d2e7a1ae8b016fb3ef
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
I20260926 05:46:55.208117 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.198262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:34013 (local address 127.11.247.62:33717)
0926 05:46:55.198325 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:55.198327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:55.198541 (+ 214us) server_negotiation.cc:407] Connection header received
0926 05:46:55.199295 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:55.199297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:55.199309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:55.199371 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:55.200354 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.200908 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.201963 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.202195 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.202290 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.204857 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:55.204859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:55.204859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:55.204868 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:55.206768 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.207192 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.207194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.207194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.207257 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.207506 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.207507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.207508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.207623 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:55.207677 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:55.207916 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 05:46:55.207990 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:46:55.208220 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.198245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33717 (local address 127.11.247.1:34013)
0926 05:46:55.198415 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:55.198453 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:55.199161 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:55.199450 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:55.199454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:55.200034 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:55.200243 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:55.200249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.201013 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.201014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:55.201866 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:55.201869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.202303 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.202305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:55.202445 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.203055 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:55.203063 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:55.204762 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:55.206851 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:55.206858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:55.206860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:55.207088 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:55.207328 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:55.207329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:55.207329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:55.207434 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:55.207681 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:55.207685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:55.207805 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 05:46:55.207887 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:46:55.208013 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260926 05:46:55.208815 15489 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33717
I20260926 05:46:55.208936 15489 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:55.209213 15489 heartbeater.cc:511] Master 127.11.247.62:33717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
I20260926 05:46:55.209792 15298 ts_manager.cc:194] Registered new tserver with Master: b75c0a73fd2342e3850030f8cff3c833 (127.11.247.1:35741)
I20260926 05:46:55.210446 15298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:34013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:55.225498 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33717
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:55.309064 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:55.309283 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:55.309309 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:55.310511 15494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:55.310559 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:55.310638 15494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:55.312144 15494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:55.312398 15494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:55.312644 15494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:55.318053 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:55.318147 15494 server_base.cc:1034] running on GCE node
W20260926 05:46:55.318037 15499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:55.318038 15500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:55.318475 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:55.318765 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:55.319901 15494 hybrid_clock.cc:648] HybridClock initialized: now 1790401615319890 us; error 32 us; skew 500 ppm
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:55.322453 15494 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:55.323004 15494 webserver.cc:460] Webserver started at http://127.11.247.2:37491/ using document root <none> and password file <none>
I20260926 05:46:55.323247 15494 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:55.323297 15494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:55.323482 15494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:55.324781 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "c4e5c770579e46f3a42d601ba5b4db3a"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "a9ec5a01957ae9d346575be61bdc766b"
server_key_iv: "84f3c51ff4c89d6a05bae8739e009c1d"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.325183 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "c4e5c770579e46f3a42d601ba5b4db3a"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "a9ec5a01957ae9d346575be61bdc766b"
server_key_iv: "84f3c51ff4c89d6a05bae8739e009c1d"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.326680 15494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260926 05:46:55.327521 15509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.327739 15494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:55.327840 15494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "c4e5c770579e46f3a42d601ba5b4db3a"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "a9ec5a01957ae9d346575be61bdc766b"
server_key_iv: "84f3c51ff4c89d6a05bae8739e009c1d"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.327917 15494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:55.338047 15494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:55.340162 15494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:55.340333 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:55.340616 15494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:55.340947 15494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:55.340986 15494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.341046 15494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:55.341081 15494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.347945 15494 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:38835
I20260926 05:46:55.347990 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:38835 every 8 connection(s)
I20260926 05:46:55.348378 15494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:55.350437 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15494
I20260926 05:46:55.350556 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:55.350807 12252 external_mini_cluster.cc:1428] Setting key 83c6702bbf50c3f96c7d71cc31f65c41
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:46:55.359561 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.349187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:45761 (local address 127.11.247.62:33717)
0926 05:46:55.349276 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:55.349279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:55.349465 (+ 186us) server_negotiation.cc:407] Connection header received
0926 05:46:55.349940 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:55.349942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:55.349953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:55.350083 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:55.351364 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.352091 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.353270 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.353581 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.353666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.356331 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:55.356333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:55.356333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:55.356341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:55.357919 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.358420 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.358421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.358422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.358488 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.358805 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.358806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.358806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.358968 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:55.359056 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:55.359365 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 05:46:55.359458 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:46:55.359763 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.349152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33717 (local address 127.11.247.2:45761)
0926 05:46:55.349338 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:55.349375 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:55.349850 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:55.350154 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:55.350158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:55.351043 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:55.351272 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:55.351278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.352184 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.352184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:55.353161 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:55.353164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.353681 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.353683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:55.353830 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.354419 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:55.354430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:55.356195 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:55.358056 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:55.358064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:55.358066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:55.358317 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:55.358567 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:55.358568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:55.358568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:55.358718 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:55.359047 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:55.359054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:55.359212 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 05:46:55.359325 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 05:46:55.359503 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260926 05:46:55.360440 15623 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33717
I20260926 05:46:55.360543 15623 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:55.360693 15623 heartbeater.cc:511] Master 127.11.247.62:33717 requested a full tablet report, sending...
I20260926 05:46:55.361099 15298 ts_manager.cc:194] Registered new tserver with Master: c4e5c770579e46f3a42d601ba5b4db3a (127.11.247.2:38835)
I20260926 05:46:55.361486 15298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:45761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:55.378384 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33717
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:55.461623 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:55.461879 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:55.461917 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:55.463203 15628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:55.463459 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:55.463543 15628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:55.465106 15628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:55.465415 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:55.465673 15628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:46:55.470815 15628 server_base.cc:1034] running on GCE node
W20260926 05:46:55.470767 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:55.470775 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:55.470808 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:55.471158 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:55.471370 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:55.472543 15628 hybrid_clock.cc:648] HybridClock initialized: now 1790401615472521 us; error 35 us; skew 500 ppm
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:55.475243 15628 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:55.475656 15628 webserver.cc:460] Webserver started at http://127.11.247.3:41145/ using document root <none> and password file <none>
I20260926 05:46:55.475837 15628 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:55.475880 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:55.475970 15628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:55.477000 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "3d0f7d6bcc354fe7a84256afe87ea1b0"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "823b3b0513dd8522d467f6fa4e273a5a"
server_key_iv: "ba38ee4ef7433dea3e3f3e816adcc8e0"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.477321 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "3d0f7d6bcc354fe7a84256afe87ea1b0"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "823b3b0513dd8522d467f6fa4e273a5a"
server_key_iv: "ba38ee4ef7433dea3e3f3e816adcc8e0"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.478714 15628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:46:55.479518 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.479676 15628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:55.479773 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "3d0f7d6bcc354fe7a84256afe87ea1b0"
format_stamp: "Formatted at 2026-09-26 05:46:55 on dist-test-slave-21p4"
server_key: "823b3b0513dd8522d467f6fa4e273a5a"
server_key_iv: "ba38ee4ef7433dea3e3f3e816adcc8e0"
server_key_version: "encryptionkey@0"
I20260926 05:46:55.479875 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:55.489286 15628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:55.491228 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:55.491394 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:55.491629 15628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:55.491945 15628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:55.492003 15628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.492053 15628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:55.492084 15628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:55.498354 15628 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:43771
I20260926 05:46:55.498390 15756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:43771 every 8 connection(s)
I20260926 05:46:55.498780 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:55.503459 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15628
I20260926 05:46:55.503578 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:55.503774 12252 external_mini_cluster.cc:1428] Setting key a811112f39f7af08fe4ddcd0640d1070
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:55.508621 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.499584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:40681 (local address 127.11.247.62:33717)
0926 05:46:55.499653 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:55.499655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:55.499847 (+ 192us) server_negotiation.cc:407] Connection header received
0926 05:46:55.500376 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:55.500378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:55.500390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:55.500449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:55.501461 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.501966 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.503155 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.503350 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.503452 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.505938 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:55.505940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:55.505940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:55.505948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:55.507272 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.507686 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.507687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.507688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.507748 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.507993 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.507994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.507995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.508108 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:55.508186 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:55.508450 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 05:46:55.508528 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:46:55.508838 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.499541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33717 (local address 127.11.247.3:40681)
0926 05:46:55.499725 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:55.499769 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:55.500269 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:55.500530 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:55.500533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:55.501168 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:55.501370 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:55.501375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.502088 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.502089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:55.503069 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:55.503072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.503471 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.503473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:55.503634 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.504261 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:55.504272 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:55.505845 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:55.507370 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:55.507377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:55.507379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:55.507601 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:55.507818 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:55.507819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:55.507819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:55.507927 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:55.508177 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:55.508183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:55.508323 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 05:46:55.508429 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:46:55.508589 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260926 05:46:55.509308 15757 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33717
I20260926 05:46:55.509419 15757 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:55.509608 15757 heartbeater.cc:511] Master 127.11.247.62:33717 requested a full tablet report, sending...
I20260926 05:46:55.509967 15298 ts_manager.cc:194] Registered new tserver with Master: 3d0f7d6bcc354fe7a84256afe87ea1b0 (127.11.247.3:43771)
I20260926 05:46:55.510351 15298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:40681
I20260926 05:46:55.515733 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401615, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:55.539153 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:55.531524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.11.247.62:33717)
0926 05:46:55.531605 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:55.531609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:55.531707 (+ 98us) server_negotiation.cc:407] Connection header received
0926 05:46:55.531809 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:55.531810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:55.531822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:55.531901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:55.532538 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.533041 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.533788 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:55.533994 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:55.534081 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:55.536378 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:55.536380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:55.536381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:55.536388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:55.537779 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.538259 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.538261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.538262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.538322 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:55.538516 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:55.538517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:55.538517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:55.538733 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:55.538842 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:55.538952 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 05:46:55.539045 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 05:46:55.539412 15298 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33082
I20260926 05:46:55.539954 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15360
I20260926 05:46:55.544346 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15494
I20260926 05:46:55.548869 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15628
I20260926 05:46:55.553110 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15266
2026-09-26T05:46:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2116 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15769](info): set up 2 sockets
Sep 26 05:46:55 dist-test-slave-21p4 krb5kdc[15769](info): commencing operation
krb5kdc: starting...
W20260926 05:46:56.606782 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:56 dist-test-slave-21p4 krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:56.727731 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:36177
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:56.813239 15782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:56.813504 15782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:56.813544 15782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:56.814774 15782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:56.814844 15782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:56.814868 15782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:56.814900 15782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:56.814930 15782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:56.816606 15782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:56.816921 15782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:56.817169 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:56.822841 15787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:56.822845 15788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:56.822841 15790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:56.822880 15782 server_base.cc:1034] running on GCE node
I20260926 05:46:56.823346 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:56.823611 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:56.824790 15782 hybrid_clock.cc:648] HybridClock initialized: now 1790401616824761 us; error 49 us; skew 500 ppm
Sep 26 05:46:56 dist-test-slave-21p4 krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401616, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:56.827739 15782 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:56.828351 15782 webserver.cc:460] Webserver started at http://127.11.247.62:40373/ using document root <none> and password file <none>
I20260926 05:46:56.828605 15782 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:56.828681 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:56.828841 15782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:56.829980 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "345985c218d6464fb4bc34514ecddd19"
format_stamp: "Formatted at 2026-09-26 05:46:56 on dist-test-slave-21p4"
server_key: "25445a7ed504d066acbf12d84bee934d"
server_key_iv: "150cbefa6849cf6fb039025e34eaa0a1"
server_key_version: "encryptionkey@0"
I20260926 05:46:56.830380 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "345985c218d6464fb4bc34514ecddd19"
format_stamp: "Formatted at 2026-09-26 05:46:56 on dist-test-slave-21p4"
server_key: "25445a7ed504d066acbf12d84bee934d"
server_key_iv: "150cbefa6849cf6fb039025e34eaa0a1"
server_key_version: "encryptionkey@0"
I20260926 05:46:56.832026 15782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:56.832890 15797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:56.833139 15782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:56.833258 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "345985c218d6464fb4bc34514ecddd19"
format_stamp: "Formatted at 2026-09-26 05:46:56 on dist-test-slave-21p4"
server_key: "25445a7ed504d066acbf12d84bee934d"
server_key_iv: "150cbefa6849cf6fb039025e34eaa0a1"
server_key_version: "encryptionkey@0"
I20260926 05:46:56.833348 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:56.851249 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:56.853442 15782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:56.853613 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:56.857784 15782 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:36177
I20260926 05:46:56.857818 15849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:36177 every 8 connection(s)
I20260926 05:46:56.858338 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:56.859028 15850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:56.861369 15850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19: Bootstrap starting.
I20260926 05:46:56.861994 15850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:56.862301 15850 log.cc:824] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:56.863131 15850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19: No bootstrap required, opened a new log
I20260926 05:46:56.863206 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15782
I20260926 05:46:56.863408 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:56.863741 12252 external_mini_cluster.cc:1428] Setting key 0f6e7054ff2efa4c869538f261c4b967
I20260926 05:46:56.865949 15850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345985c218d6464fb4bc34514ecddd19" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36177 } }
I20260926 05:46:56.866132 15850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:56.866216 15850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345985c218d6464fb4bc34514ecddd19, State: Initialized, Role: FOLLOWER
I20260926 05:46:56.866405 15850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345985c218d6464fb4bc34514ecddd19" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36177 } }
I20260926 05:46:56.866518 15850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:56.866552 15850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:56.866629 15850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:56.867362 15850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345985c218d6464fb4bc34514ecddd19" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36177 } }
I20260926 05:46:56.867451 15850 leader_election.cc:302] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 345985c218d6464fb4bc34514ecddd19; no voters:
I20260926 05:46:56.867646 15850 leader_election.cc:288] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:56.867702 15855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:56.867915 15855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [term 1 LEADER]: Becoming Leader. State: Replica: 345985c218d6464fb4bc34514ecddd19, State: Running, Role: LEADER
I20260926 05:46:56.867921 15850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:56.868077 15855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345985c218d6464fb4bc34514ecddd19" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36177 } }
I20260926 05:46:56.868484 15857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 345985c218d6464fb4bc34514ecddd19. Latest consensus state: current_term: 1 leader_uuid: "345985c218d6464fb4bc34514ecddd19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345985c218d6464fb4bc34514ecddd19" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36177 } } }
I20260926 05:46:56.868494 15856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "345985c218d6464fb4bc34514ecddd19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345985c218d6464fb4bc34514ecddd19" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36177 } } }
I20260926 05:46:56.868595 15857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:56.868611 15856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:56.869498 15863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:56.870056 15863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:56.871783 15863 catalog_manager.cc:1347] Generated new cluster ID: 12b448fca13e4d5eb3296cb88938472e
I20260926 05:46:56.871840 15863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:46:56 dist-test-slave-21p4 krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:56.877395 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:56.864330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.11.247.62:36177)
0926 05:46:56.864980 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:56.864989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:56.865019 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:46:56.865948 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:56.865956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:56.866013 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:56.866215 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:56.867058 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:56.868377 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:56.869431 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:56.870088 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:56.870238 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:56.873276 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:56.873281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:56.873281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:56.873294 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:56.875415 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:56.875894 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:56.875898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:56.875899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:56.876069 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:56.876300 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:56.876301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:56.876301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:56.876641 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:56.876816 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:56.876891 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 05:46:56.877167 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":89,"threads_started":1}
I20260926 05:46:56.884300 15863 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:56.884874 15863 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:56.896350 15863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 345985c218d6464fb4bc34514ecddd19: Generated new TSK 0
I20260926 05:46:56.896613 15863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:56.926110 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36177
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:57.011037 15876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:57.011255 15876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:57.011278 15876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:57.012467 15876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:57.012509 15876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:57.012537 15876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:57.014048 15876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:57.014308 15876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:57.014592 15876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:57.019645 15881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:57.019649 15884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:57.019830 15876 server_base.cc:1034] running on GCE node
W20260926 05:46:57.019690 15882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:57.020174 15876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:57.020388 15876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:57.021529 15876 hybrid_clock.cc:648] HybridClock initialized: now 1790401617021516 us; error 28 us; skew 500 ppm
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:57.024128 15876 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:57.024627 15876 webserver.cc:460] Webserver started at http://127.11.247.1:46563/ using document root <none> and password file <none>
I20260926 05:46:57.024858 15876 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:57.024904 15876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:57.025072 15876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:57.026140 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "d1428148a9fe47b9a7320a803ebbcea9"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "6caefd4bfe4bcae3d6e8ad68d15fd0c1"
server_key_iv: "eecc16f85b2e7325c23a861118afa004"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.026523 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "d1428148a9fe47b9a7320a803ebbcea9"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "6caefd4bfe4bcae3d6e8ad68d15fd0c1"
server_key_iv: "eecc16f85b2e7325c23a861118afa004"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.027897 15876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:57.028694 15891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.028874 15876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:57.028975 15876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "d1428148a9fe47b9a7320a803ebbcea9"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "6caefd4bfe4bcae3d6e8ad68d15fd0c1"
server_key_iv: "eecc16f85b2e7325c23a861118afa004"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.029063 15876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:57.040663 15876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:57.043059 15876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:57.043238 15876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:57.043489 15876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:57.043809 15876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:57.043864 15876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.043905 15876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:57.043933 15876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.049705 15876 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:42639
I20260926 05:46:57.049786 16004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:42639 every 8 connection(s)
I20260926 05:46:57.050102 15876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:57.051363 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 15876
I20260926 05:46:57.051450 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:57.051648 12252 external_mini_cluster.cc:1428] Setting key 4684d761d461e0c9fcc28742fb75faeb
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
I20260926 05:46:57.060494 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.050869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:46887 (local address 127.11.247.62:36177)
0926 05:46:57.050935 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:57.050938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:57.051195 (+ 257us) server_negotiation.cc:407] Connection header received
0926 05:46:57.052185 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:57.052186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:57.052200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:57.052263 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:57.052993 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.053529 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.054568 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.054833 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.054919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.057475 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:57.057477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:57.057477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:57.057485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:57.059021 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.059439 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.059441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.059441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.059503 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.059852 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.059853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.059853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.059963 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:57.060018 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:57.060308 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 05:46:57.060394 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:46:57.060714 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.050871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:36177 (local address 127.11.247.1:46887)
0926 05:46:57.051028 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:57.051065 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:57.052091 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:57.052417 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:57.052421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:57.052685 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:57.052880 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:57.052884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.053628 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.053629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:57.054464 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:57.054467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.054931 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.054932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:57.055078 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.055724 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:57.055733 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:57.057369 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:57.059118 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:57.059126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:57.059128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:57.059360 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:57.059606 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:57.059608 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:57.059608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:57.059778 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:57.060024 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:57.060031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:57.060180 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 05:46:57.060272 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:46:57.060411 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260926 05:46:57.061292 16005 heartbeater.cc:347] Connected to a master server at 127.11.247.62:36177
I20260926 05:46:57.061378 16005 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:57.061571 16005 heartbeater.cc:511] Master 127.11.247.62:36177 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
I20260926 05:46:57.062289 15814 ts_manager.cc:194] Registered new tserver with Master: d1428148a9fe47b9a7320a803ebbcea9 (127.11.247.1:42639)
I20260926 05:46:57.062999 15814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:46887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:57.077888 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36177
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:57.163162 16010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:57.163384 16010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:57.163410 16010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:57.164665 16010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:57.164712 16010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:57.164739 16010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:57.166280 16010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:57.166556 16010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:57.166834 16010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:57.172459 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:57.172488 16015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:57.172477 16016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:57.172518 16010 server_base.cc:1034] running on GCE node
I20260926 05:46:57.172896 16010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:57.173142 16010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:57.174324 16010 hybrid_clock.cc:648] HybridClock initialized: now 1790401617174317 us; error 38 us; skew 500 ppm
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:57.176858 16010 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:57.177313 16010 webserver.cc:460] Webserver started at http://127.11.247.2:45791/ using document root <none> and password file <none>
I20260926 05:46:57.177567 16010 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:57.177637 16010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:57.177788 16010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:57.178959 16010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "f699a274b8a745729cdcc6de0ba22683"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "1df6f50b298902d2e1dc84fc7ccc16b0"
server_key_iv: "270a8d91b22a35978d136d5ccbd2f763"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.179323 16010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "f699a274b8a745729cdcc6de0ba22683"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "1df6f50b298902d2e1dc84fc7ccc16b0"
server_key_iv: "270a8d91b22a35978d136d5ccbd2f763"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.180670 16010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:57.181427 16025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.181591 16010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:57.181694 16010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "f699a274b8a745729cdcc6de0ba22683"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "1df6f50b298902d2e1dc84fc7ccc16b0"
server_key_iv: "270a8d91b22a35978d136d5ccbd2f763"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.181770 16010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:57.189664 16010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:57.191515 16010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:57.191682 16010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:57.191912 16010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:57.192209 16010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:57.192266 16010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.192306 16010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:57.192333 16010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.197944 16010 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:42139
I20260926 05:46:57.198017 16138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:42139 every 8 connection(s)
I20260926 05:46:57.198343 16010 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:57.203806 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16010
I20260926 05:46:57.203933 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:57.204134 12252 external_mini_cluster.cc:1428] Setting key 37dcdf2103a328f8cbf6aed656e63c9a
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:57.209494 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.199118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:39217 (local address 127.11.247.62:36177)
0926 05:46:57.199188 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:57.199190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:57.199424 (+ 234us) server_negotiation.cc:407] Connection header received
0926 05:46:57.200395 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:57.200397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:57.200410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:57.200481 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:57.201123 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.201700 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.202865 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.203140 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.203209 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.205900 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:57.205902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:57.205902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:57.205915 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:57.207802 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.208286 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.208288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.208288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.208362 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.208615 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.208616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.208617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.208797 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:57.208879 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:57.209249 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 05:46:57.209363 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:46:57.209517 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.199096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:36177 (local address 127.11.247.2:39217)
0926 05:46:57.199272 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:57.199314 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:57.200256 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:57.200540 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:57.200544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:57.200819 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:57.201022 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:57.201027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.201797 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.201797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:57.202770 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:57.202773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.203227 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.203228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:57.203344 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.203902 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:57.203916 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:57.205779 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:57.207911 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:57.207921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:57.207923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:57.208198 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:57.208433 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:57.208433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:57.208434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:57.208545 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:57.208869 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:57.208874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:57.209073 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 05:46:57.209191 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 05:46:57.209334 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260926 05:46:57.210148 16139 heartbeater.cc:347] Connected to a master server at 127.11.247.62:36177
I20260926 05:46:57.210290 16139 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:57.210565 16139 heartbeater.cc:511] Master 127.11.247.62:36177 requested a full tablet report, sending...
I20260926 05:46:57.211138 15814 ts_manager.cc:194] Registered new tserver with Master: f699a274b8a745729cdcc6de0ba22683 (127.11.247.2:42139)
I20260926 05:46:57.211530 15814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:39217
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:57.234491 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36177
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:57.318430 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:57.318720 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:57.318768 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:57.319975 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:57.320051 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:57.320096 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:57.321652 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:57.321939 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:57.322175 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:57.327514 16149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:57.327522 16150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:57.327724 16144 server_base.cc:1034] running on GCE node
W20260926 05:46:57.327540 16152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:57.328030 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:57.328339 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:57.329545 16144 hybrid_clock.cc:648] HybridClock initialized: now 1790401617329534 us; error 34 us; skew 500 ppm
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:57.332425 16144 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:57.332882 16144 webserver.cc:460] Webserver started at http://127.11.247.3:34977/ using document root <none> and password file <none>
I20260926 05:46:57.333148 16144 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:57.333215 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:57.333357 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:57.334465 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "8e3002897463482ba884a8d020b09bf6"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "f1ef14089cbfd77ca22d842a766c8f1b"
server_key_iv: "c2d51d2756f355aae52a1290b00692df"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.334942 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "8e3002897463482ba884a8d020b09bf6"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "f1ef14089cbfd77ca22d842a766c8f1b"
server_key_iv: "c2d51d2756f355aae52a1290b00692df"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.336380 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:57.337206 16159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.337483 16144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:57.337580 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "8e3002897463482ba884a8d020b09bf6"
format_stamp: "Formatted at 2026-09-26 05:46:57 on dist-test-slave-21p4"
server_key: "f1ef14089cbfd77ca22d842a766c8f1b"
server_key_iv: "c2d51d2756f355aae52a1290b00692df"
server_key_version: "encryptionkey@0"
I20260926 05:46:57.337673 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:57.349061 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:57.351557 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:57.351718 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:57.351958 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:57.352280 16144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:57.352332 16144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.352376 16144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:57.352402 16144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:57.358362 16144 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:46675
I20260926 05:46:57.358407 16272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:46675 every 8 connection(s)
I20260926 05:46:57.358798 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:57.359699 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16144
I20260926 05:46:57.359819 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:57.360049 12252 external_mini_cluster.cc:1428] Setting key dbc53e22b695fd568807ae005c46a531
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:57.369977 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.359726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:46439 (local address 127.11.247.62:36177)
0926 05:46:57.359812 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:57.359815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:57.360005 (+ 190us) server_negotiation.cc:407] Connection header received
0926 05:46:57.361204 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:57.361205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:57.361215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:57.361264 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:57.361913 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.362431 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.363606 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.363813 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.363880 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.366501 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:57.366503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:57.366503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:57.366510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:57.368219 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.368779 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.368780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.368808 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.368880 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.369210 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.369211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.369212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.369404 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:57.369499 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:57.369792 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 05:46:57.369882 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:46:57.370213 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.359641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:36177 (local address 127.11.247.3:46439)
0926 05:46:57.359865 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:57.359910 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:57.361111 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:57.361324 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:57.361328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:57.361608 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:57.361807 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:57.361812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.362548 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.362549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:57.363498 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:57.363500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.363903 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.363904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:57.364040 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.364567 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:57.364577 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:57.366352 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:57.368349 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:57.368361 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:57.368364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:57.368693 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:57.369009 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:57.369009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:57.369010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:57.369133 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:57.369491 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:57.369499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:57.369646 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 05:46:57.369754 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 05:46:57.369930 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260926 05:46:57.370716 16273 heartbeater.cc:347] Connected to a master server at 127.11.247.62:36177
I20260926 05:46:57.370813 16273 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:57.371017 16273 heartbeater.cc:511] Master 127.11.247.62:36177 requested a full tablet report, sending...
I20260926 05:46:57.371484 15813 ts_manager.cc:194] Registered new tserver with Master: 8e3002897463482ba884a8d020b09bf6 (127.11.247.3:46675)
I20260926 05:46:57.371901 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:46:57.371941 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:46439
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:46:57.396404 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.387997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.11.247.1:42639)
0926 05:46:57.388151 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:57.388158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:57.388179 (+ 21us) server_negotiation.cc:407] Connection header received
0926 05:46:57.388207 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:57.388209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:57.388223 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:57.388292 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:57.388877 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.389607 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.390340 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.390765 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.390824 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.393342 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:57.393352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:57.393352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:57.393358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:57.394989 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.395365 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.395369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.395369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.395431 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.395629 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.395630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.395631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.395797 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:57.396008 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:57.396062 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:46:57.396223 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[15769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:57.404641 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:57.397727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.11.247.62:36177)
0926 05:46:57.397796 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:57.397798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:57.397807 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:57.397870 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:57.397871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:57.397881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:57.397921 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:57.398425 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.398984 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.399667 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:57.399849 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:57.399904 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:57.402137 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:57.402139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:57.402139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:57.402171 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:57.403508 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.403833 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.403834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.403834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.403881 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:57.404129 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:57.404129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:57.404130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:57.404282 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:57.404360 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:57.404469 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 05:46:57.404531 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 05:46:57.404950 15813 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39030
I20260926 05:46:57.405347 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15876
I20260926 05:46:57.410130 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16010
I20260926 05:46:57.414234 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16144
I20260926 05:46:57.418406 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 15782
2026-09-26T05:46:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1864 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[16285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[16285](info): set up 2 sockets
Sep 26 05:46:57 dist-test-slave-21p4 krb5kdc[16285](info): commencing operation
krb5kdc: starting...
W20260926 05:46:58.467056 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:46:58 dist-test-slave-21p4 krb5kdc[16285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:46:58.591112 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40285
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:58.674633 16298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:58.674901 16298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:58.674939 16298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:58.676139 16298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:58.676214 16298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:58.676239 16298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:58.676259 16298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:58.676286 16298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:58.677855 16298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:40285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:58.678153 16298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:58.678398 16298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:58.683749 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:58.683707 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:58.683748 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:58.684001 16298 server_base.cc:1034] running on GCE node
I20260926 05:46:58.684250 16298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:58.684554 16298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:58.685750 16298 hybrid_clock.cc:648] HybridClock initialized: now 1790401618685744 us; error 42 us; skew 500 ppm
Sep 26 05:46:58 dist-test-slave-21p4 krb5kdc[16285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:58.694346 16298 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:46:58.694849 16298 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:58.694931 16298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:58.695102 16298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:58.696247 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "4454843947b144efb39202e287dda19c"
format_stamp: "Formatted at 2026-09-26 05:46:58 on dist-test-slave-21p4"
server_key: "70f3ac910191aa811976283a5ab26fd2"
server_key_iv: "d52e309fff099f565569bbfb3b76c9f5"
server_key_version: "encryptionkey@0"
I20260926 05:46:58.696622 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "4454843947b144efb39202e287dda19c"
format_stamp: "Formatted at 2026-09-26 05:46:58 on dist-test-slave-21p4"
server_key: "70f3ac910191aa811976283a5ab26fd2"
server_key_iv: "d52e309fff099f565569bbfb3b76c9f5"
server_key_version: "encryptionkey@0"
I20260926 05:46:58.698036 16298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:58.698948 16312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:58.699215 16298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:58.699337 16298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "4454843947b144efb39202e287dda19c"
format_stamp: "Formatted at 2026-09-26 05:46:58 on dist-test-slave-21p4"
server_key: "70f3ac910191aa811976283a5ab26fd2"
server_key_iv: "d52e309fff099f565569bbfb3b76c9f5"
server_key_version: "encryptionkey@0"
I20260926 05:46:58.699424 16298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:58.724341 16298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:58.726854 16298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:58.727037 16298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:58.731460 16298 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:40285
I20260926 05:46:58.731489 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:40285 every 8 connection(s)
I20260926 05:46:58.731793 16298 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:58.732752 16365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:58.735189 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c: Bootstrap starting.
I20260926 05:46:58.735780 16365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:58.736137 16365 log.cc:824] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:58.736577 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16298
I20260926 05:46:58.736693 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:58.736889 12252 external_mini_cluster.cc:1428] Setting key 5ad986bb2bbb80ab335c0210709845f8
I20260926 05:46:58.737039 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c: No bootstrap required, opened a new log
I20260926 05:46:58.739588 16365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4454843947b144efb39202e287dda19c" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40285 } }
I20260926 05:46:58.739765 16365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:58.739816 16365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4454843947b144efb39202e287dda19c, State: Initialized, Role: FOLLOWER
I20260926 05:46:58.739956 16365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4454843947b144efb39202e287dda19c" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40285 } }
I20260926 05:46:58.740036 16365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:58.740080 16365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:58.740127 16365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:58.740692 16365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4454843947b144efb39202e287dda19c" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40285 } }
I20260926 05:46:58.740813 16365 leader_election.cc:302] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4454843947b144efb39202e287dda19c; no voters:
I20260926 05:46:58.741009 16365 leader_election.cc:288] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:58.741037 16370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:58.741164 16370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [term 1 LEADER]: Becoming Leader. State: Replica: 4454843947b144efb39202e287dda19c, State: Running, Role: LEADER
I20260926 05:46:58.741295 16365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:58.741312 16370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4454843947b144efb39202e287dda19c" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40285 } }
I20260926 05:46:58.741623 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4454843947b144efb39202e287dda19c. Latest consensus state: current_term: 1 leader_uuid: "4454843947b144efb39202e287dda19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4454843947b144efb39202e287dda19c" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40285 } } }
I20260926 05:46:58.741637 16372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4454843947b144efb39202e287dda19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4454843947b144efb39202e287dda19c" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40285 } } }
I20260926 05:46:58.741703 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:58.741748 16372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:58.742787 16383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:58.743356 16383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:46:58 dist-test-slave-21p4 krb5kdc[16285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:58.745028 16383 catalog_manager.cc:1347] Generated new cluster ID: 24ca5987c3d4496ebc70a278d88f6be4
I20260926 05:46:58.745083 16383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:46:58.748849 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:58.737483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.11.247.62:40285)
0926 05:46:58.737660 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:58.737674 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:58.737702 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:46:58.738336 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:58.738343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:58.738395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:58.738630 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:58.739280 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:58.740444 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:58.741436 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:58.742149 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:58.742242 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:58.745426 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:58.745430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:58.745431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:58.745443 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:58.747236 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:58.747655 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:58.747658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:58.747659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:58.747753 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:58.747973 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:58.747974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:58.747974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:58.748224 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:58.748391 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:58.748487 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:46:58.748631 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260926 05:46:58.768502 16383 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:58.769345 16383 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:58.772949 16383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c: Generated new TSK 0
I20260926 05:46:58.773162 16383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:46:58.806929 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40285
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:58.891644 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:58.891903 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:58.891948 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:58.893211 16391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:58.893289 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:58.893338 16391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:58.895069 16391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:58.895351 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:58.895605 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:58.900736 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:58.900843 16397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:58.900833 16396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:58.901106 16391 server_base.cc:1034] running on GCE node
I20260926 05:46:58.901333 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:58.901599 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:58.902767 16391 hybrid_clock.cc:648] HybridClock initialized: now 1790401618902751 us; error 46 us; skew 500 ppm
Sep 26 05:46:58 dist-test-slave-21p4 krb5kdc[16285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:58.905407 16391 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:46:58.905810 16391 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:58.905860 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:58.905952 16391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:58.907083 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "4eacb9199f124c8b91fdb02a3445bac6"
format_stamp: "Formatted at 2026-09-26 05:46:58 on dist-test-slave-21p4"
server_key: "cb1501867fd3dbd5413c1c3260186007"
server_key_iv: "e8f89880ceea982421e4c654aec33784"
server_key_version: "encryptionkey@0"
I20260926 05:46:58.907388 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "4eacb9199f124c8b91fdb02a3445bac6"
format_stamp: "Formatted at 2026-09-26 05:46:58 on dist-test-slave-21p4"
server_key: "cb1501867fd3dbd5413c1c3260186007"
server_key_iv: "e8f89880ceea982421e4c654aec33784"
server_key_version: "encryptionkey@0"
I20260926 05:46:58.908676 16391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:58.909399 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:58.909629 16391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:58.909718 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "4eacb9199f124c8b91fdb02a3445bac6"
format_stamp: "Formatted at 2026-09-26 05:46:58 on dist-test-slave-21p4"
server_key: "cb1501867fd3dbd5413c1c3260186007"
server_key_iv: "e8f89880ceea982421e4c654aec33784"
server_key_version: "encryptionkey@0"
I20260926 05:46:58.909775 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:58.917408 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:58.919670 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:58.919792 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:58.920008 16391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:58.920392 16391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:58.920428 16391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:58.920450 16391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:58.920462 16391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:58.926198 16391 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:45133
I20260926 05:46:58.926254 16518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:45133 every 8 connection(s)
I20260926 05:46:58.926533 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:58.932474 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16391
I20260926 05:46:58.932619 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:58.932825 12252 external_mini_cluster.cc:1428] Setting key e13f2bac55f9f1ff6b1636184a324a2d
Sep 26 05:46:58 dist-test-slave-21p4 krb5kdc[16285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:58.936944 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:58.927414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:56999 (local address 127.11.247.62:40285)
0926 05:46:58.927491 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:58.927494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:58.927742 (+ 248us) server_negotiation.cc:407] Connection header received
0926 05:46:58.928217 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:58.928218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:58.928230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:58.928290 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:58.929265 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:58.929842 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:58.930947 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:58.931178 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:58.931244 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:58.933713 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:58.933715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:58.933716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:58.933724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:58.935381 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:58.935795 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:58.935797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:58.935798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:58.935872 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:58.936163 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:58.936164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:58.936164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:58.936325 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:58.936402 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:58.936674 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 05:46:58.936789 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:46:58.936939 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:58.927413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40285 (local address 127.11.247.1:56999)
0926 05:46:58.927610 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:58.927648 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:58.928120 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:58.928363 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:58.928367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:58.928639 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:58.929145 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:58.929151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:58.929937 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:58.929938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:58.930856 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:58.930859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:58.931257 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:58.931258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:58.931368 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:58.931906 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:58.931915 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:58.933612 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:58.935477 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:58.935484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:58.935486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:58.935713 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:58.935941 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:58.935942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:58.935942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:58.936092 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:58.936396 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:58.936402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:58.936536 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 05:46:58.936625 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:46:58.936762 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":108,"threads_started":1}
I20260926 05:46:58.937525 16519 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40285
I20260926 05:46:58.937645 16519 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:58.937819 16519 heartbeater.cc:511] Master 127.11.247.62:40285 requested a full tablet report, sending...
I20260926 05:46:58.938449 16329 ts_manager.cc:194] Registered new tserver with Master: 4eacb9199f124c8b91fdb02a3445bac6 (127.11.247.1:45133)
I20260926 05:46:58.939561 16329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:56999
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:46:58.961592 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40285
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:59.045644 16524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:59.045862 16524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:59.045887 16524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:59.047149 16524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:59.047197 16524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:59.047225 16524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:59.048673 16524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:59.048870 16524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:59.049059 16524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:59.054273 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.054306 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.054435 16524 server_base.cc:1034] running on GCE node
W20260926 05:46:59.054277 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.054762 16524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:59.055018 16524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:59.056212 16524 hybrid_clock.cc:648] HybridClock initialized: now 1790401619056183 us; error 46 us; skew 500 ppm
Sep 26 05:46:59 dist-test-slave-21p4 krb5kdc[16285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401619, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:59.059101 16524 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:46:59.059480 16524 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:59.059532 16524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:59.059626 16524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:59.060671 16524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "da0d76e21e7704284289551d6a232736"
server_key_iv: "3b52b52deb40e93510bb1e216238ab32"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.060997 16524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "da0d76e21e7704284289551d6a232736"
server_key_iv: "3b52b52deb40e93510bb1e216238ab32"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.062314 16524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:46:59.063313 16538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.063571 16524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:46:59.063668 16524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "da0d76e21e7704284289551d6a232736"
server_key_iv: "3b52b52deb40e93510bb1e216238ab32"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.063730 16524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:59.080785 16524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:59.083201 16524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:59.083328 16524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:59.083525 16524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:59.083868 16524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:59.083904 16524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.083930 16524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:59.083941 16524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.090204 16524 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35971
I20260926 05:46:59.090233 16651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35971 every 8 connection(s)
I20260926 05:46:59.090528 16524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:46:59 dist-test-slave-21p4 krb5kdc[16285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401619, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:59.097204 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16524
I20260926 05:46:59.097303 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:59.097527 12252 external_mini_cluster.cc:1428] Setting key f0275cc8345d2e0268a37f3740090d1c
I20260926 05:46:59.100628 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.091329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:59873 (local address 127.11.247.62:40285)
0926 05:46:59.091376 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.091379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.091604 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:46:59.092066 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.092067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.092076 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.092141 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:59.093083 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.093589 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.094691 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.094888 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.094952 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.097154 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:59.097156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:59.097156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:59.097164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:59.098832 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:59.099341 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:59.099342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:59.099342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:59.099393 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:59.099743 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:59.099744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:59.099745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:59.099943 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:59.100031 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.100426 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.100521 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:46:59.100843 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.091285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40285 (local address 127.11.247.2:59873)
0926 05:46:59.091449 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.091494 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.091983 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.092197 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.092200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.092461 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:59.092998 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.093004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.093682 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.093683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.094505 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.094507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.094968 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.094968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.095087 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.095580 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:59.095589 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:59.097018 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:59.098909 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:59.098917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:59.098920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:59.099259 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:59.099453 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:59.099454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:59.099454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:59.099624 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:59.100049 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:59.100057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:59.100257 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.100377 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.100512 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260926 05:46:59.101464 16652 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40285
I20260926 05:46:59.101572 16652 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:59.101734 16652 heartbeater.cc:511] Master 127.11.247.62:40285 requested a full tablet report, sending...
I20260926 05:46:59.102218 16329 ts_manager.cc:194] Registered new tserver with Master: 0d434ba3524b4e57a1a5e4f42e89e6a2 (127.11.247.2:35971)
I20260926 05:46:59.102711 16329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:59873
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:46:59.125659 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40285
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:59.213794 16657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:59.214016 16657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:59.214042 16657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:59.215236 16657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:59.215281 16657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:59.215309 16657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:46:59.216760 16657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:59.216957 16657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:59.217137 16657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:59.222410 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.222442 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.222455 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.222681 16657 server_base.cc:1034] running on GCE node
I20260926 05:46:59.222929 16657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:59.223177 16657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:59.224370 16657 hybrid_clock.cc:648] HybridClock initialized: now 1790401619224351 us; error 27 us; skew 500 ppm
Sep 26 05:46:59 dist-test-slave-21p4 krb5kdc[16285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401619, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:59.227291 16657 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:46:59.227676 16657 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:59.227726 16657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:59.227821 16657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:59.228929 16657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "589f8dac7e4a410a8c2ec47cfacf56fe"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "17bcc06ee80dd2a554d884c362c78404"
server_key_iv: "f7bcd2eac7bf2b99dcceffb36389ace7"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.229287 16657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "589f8dac7e4a410a8c2ec47cfacf56fe"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "17bcc06ee80dd2a554d884c362c78404"
server_key_iv: "f7bcd2eac7bf2b99dcceffb36389ace7"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.230700 16657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:46:59.231528 16671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.231729 16657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:59.231823 16657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "589f8dac7e4a410a8c2ec47cfacf56fe"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "17bcc06ee80dd2a554d884c362c78404"
server_key_iv: "f7bcd2eac7bf2b99dcceffb36389ace7"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.231884 16657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:59.259658 16657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:59.262216 16657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:59.262336 16657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:59.262537 16657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:59.262954 16657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:59.262993 16657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.263018 16657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:59.263031 16657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.269150 16657 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:35815
I20260926 05:46:59.269232 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:35815 every 8 connection(s)
I20260926 05:46:59.269538 16657 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:46:59.271443 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16657
I20260926 05:46:59.271576 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:46:59.271754 12252 external_mini_cluster.cc:1428] Setting key 3d96ea44c227f88f7ef2aee948edae2e
Sep 26 05:46:59 dist-test-slave-21p4 krb5kdc[16285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401619, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:46:59.280058 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.270368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45547 (local address 127.11.247.62:40285)
0926 05:46:59.270447 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.270449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.270752 (+ 303us) server_negotiation.cc:407] Connection header received
0926 05:46:59.271324 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.271326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.271339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.271396 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:59.272591 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.273265 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.274328 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.274546 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.274648 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.277086 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:59.277087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:59.277088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:59.277097 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:59.278565 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:59.279049 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:59.279051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:59.279051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:59.279114 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:59.279378 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:59.279379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:59.279379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:59.279509 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:59.279580 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.279879 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.279958 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 05:46:59.280263 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.270379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40285 (local address 127.11.247.3:45547)
0926 05:46:59.270555 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.270656 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.271202 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.271483 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.271488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.271862 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:59.272487 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.272494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.273379 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.273380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.274216 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.274219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.274656 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.274657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.274795 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.275360 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:59.275368 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:59.276935 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:59.278728 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:59.278734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:59.278737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:59.278960 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:59.279181 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:59.279182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:59.279182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:59.279302 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:59.279582 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:59.279589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:59.279737 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.279847 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.280020 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260926 05:46:59.280784 16785 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40285
I20260926 05:46:59.280900 16785 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:59.281097 16785 heartbeater.cc:511] Master 127.11.247.62:40285 requested a full tablet report, sending...
I20260926 05:46:59.281535 16329 ts_manager.cc:194] Registered new tserver with Master: 589f8dac7e4a410a8c2ec47cfacf56fe (127.11.247.3:35815)
I20260926 05:46:59.281877 16329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:45547
I20260926 05:46:59.283744 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:46:59.291080 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.285109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.11.247.62:40285)
0926 05:46:59.285173 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.285174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.285214 (+ 40us) server_negotiation.cc:407] Connection header received
0926 05:46:59.285262 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.285263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.285276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.285329 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:59.285845 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.286351 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.287157 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.287342 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.287402 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.288494 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:59.288495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:59.288495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:59.288501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:59.289900 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:59.290340 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:59.290341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:59.290342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:59.290392 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:59.290560 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:59.290560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:59.290561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:59.290776 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:59.290857 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.290955 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.291014 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:46:59.292935 16329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:46:59.294641 16329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:46:59.302361 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.297656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.11.247.3:35815)
0926 05:46:59.298180 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.298185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.298195 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:59.298234 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.298236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.298251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.298332 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:59.298560 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.299686 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.301068 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.301822 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.301896 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.301958 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.302204 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.302258 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":86,"threads_started":1}
I20260926 05:46:59.302376 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.297426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:35815 (local address 127.0.0.1:34358)
0926 05:46:59.297591 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.297640 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.297712 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.298425 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.298426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.298428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:59.298474 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.298475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.299804 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.299805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.300981 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.300983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.301998 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.301999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.302089 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.302104 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.302184 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.302272 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260926 05:46:59.302388 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.297426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:45133 (local address 127.0.0.1:60452)
0926 05:46:59.297738 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.297749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.297790 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.298226 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.298229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.298232 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:59.298319 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.298322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.299456 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.299458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.300903 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.300906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.302050 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.302051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.302101 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.302107 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.302181 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.302271 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":41,"thread_start_us":98,"threads_started":1}
I20260926 05:46:59.302759 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.297558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.11.247.1:45133)
0926 05:46:59.297890 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.297896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.297910 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:46:59.297959 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.297962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.297981 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.298092 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:59.298412 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.299340 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.300997 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.301822 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.301896 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.301958 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.302568 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.302649 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":93,"threads_started":1}
I20260926 05:46:59.303452 16719 tablet_service.cc:1467] Processing CreateTablet for tablet de308913721543a2842b826fe7a7401c (DEFAULT_TABLE table=test-table [id=08ad4d1b2a974ec1aa518ba72197a094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:59.303449 16453 tablet_service.cc:1467] Processing CreateTablet for tablet de308913721543a2842b826fe7a7401c (DEFAULT_TABLE table=test-table [id=08ad4d1b2a974ec1aa518ba72197a094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:59.303826 16453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de308913721543a2842b826fe7a7401c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:59.303828 16719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de308913721543a2842b826fe7a7401c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:59.304131 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.297684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35971 (local address 127.0.0.1:44412)
0926 05:46:59.297910 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.297923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.297978 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.298310 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.298311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.298313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:59.298363 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.298365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.301077 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.301077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.301993 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.302019 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.303829 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.303830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.303945 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.303950 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.304013 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.304058 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":68,"threads_started":1}
I20260926 05:46:59.304149 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.297914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.11.247.2:35971)
0926 05:46:59.298062 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.298067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.298077 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:59.298118 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.298120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.298137 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.298214 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:59.298445 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.299704 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.302111 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.303469 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.303837 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.303916 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.304028 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.304071 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 05:46:59.304916 16586 tablet_service.cc:1467] Processing CreateTablet for tablet de308913721543a2842b826fe7a7401c (DEFAULT_TABLE table=test-table [id=08ad4d1b2a974ec1aa518ba72197a094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:46:59.305284 16586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de308913721543a2842b826fe7a7401c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:59.307262 16802 tablet_bootstrap.cc:492] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: Bootstrap starting.
I20260926 05:46:59.307966 16802 tablet_bootstrap.cc:654] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:59.308452 16804 tablet_bootstrap.cc:492] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Bootstrap starting.
I20260926 05:46:59.308598 16802 log.cc:824] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:59.308945 16803 tablet_bootstrap.cc:492] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Bootstrap starting.
I20260926 05:46:59.309093 16804 tablet_bootstrap.cc:654] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:59.309466 16803 tablet_bootstrap.cc:654] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:59.309638 16804 log.cc:824] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:59.309937 16803 log.cc:824] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:59.310456 16804 tablet_bootstrap.cc:492] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: No bootstrap required, opened a new log
I20260926 05:46:59.310484 16802 tablet_bootstrap.cc:492] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: No bootstrap required, opened a new log
I20260926 05:46:59.310534 16804 ts_tablet_manager.cc:1397] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:59.310559 16802 ts_tablet_manager.cc:1397] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:46:59.311983 16802 raft_consensus.cc:348] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.311966 16804 raft_consensus.cc:348] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.312152 16802 raft_consensus.cc:374] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:59.312170 16804 raft_consensus.cc:374] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:59.312177 16802 raft_consensus.cc:729] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eacb9199f124c8b91fdb02a3445bac6, State: Initialized, Role: FOLLOWER
I20260926 05:46:59.312218 16804 raft_consensus.cc:729] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d434ba3524b4e57a1a5e4f42e89e6a2, State: Initialized, Role: FOLLOWER
I20260926 05:46:59.312315 16802 consensus_queue.cc:260] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.312340 16804 consensus_queue.cc:260] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.312628 16804 ts_tablet_manager.cc:1428] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:59.312629 16802 ts_tablet_manager.cc:1428] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:59.312631 16652 heartbeater.cc:503] Master 127.11.247.62:40285 was elected leader, sending a full tablet report...
I20260926 05:46:59.312631 16519 heartbeater.cc:503] Master 127.11.247.62:40285 was elected leader, sending a full tablet report...
I20260926 05:46:59.313337 16803 tablet_bootstrap.cc:492] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: No bootstrap required, opened a new log
I20260926 05:46:59.313613 16803 ts_tablet_manager.cc:1397] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 05:46:59.315305 16803 raft_consensus.cc:348] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.315436 16803 raft_consensus.cc:374] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:59.315459 16803 raft_consensus.cc:729] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589f8dac7e4a410a8c2ec47cfacf56fe, State: Initialized, Role: FOLLOWER
I20260926 05:46:59.315583 16803 consensus_queue.cc:260] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.315864 16803 ts_tablet_manager.cc:1428] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:46:59.315871 16785 heartbeater.cc:503] Master 127.11.247.62:40285 was elected leader, sending a full tablet report...
W20260926 05:46:59.341163 16653 tablet.cc:2369] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:46:59.427461 16520 tablet.cc:2369] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:46:59.486150 16809 raft_consensus.cc:482] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:46:59.486339 16809 raft_consensus.cc:504] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.486804 16809 leader_election.cc:288] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589f8dac7e4a410a8c2ec47cfacf56fe (127.11.247.3:35815), 0d434ba3524b4e57a1a5e4f42e89e6a2 (127.11.247.2:35971)
I20260926 05:46:59.489774 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.487087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:40875 (local address 127.11.247.3:35815)
0926 05:46:59.487139 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.487142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.487190 (+ 48us) server_negotiation.cc:407] Connection header received
0926 05:46:59.487308 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.487309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.487326 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.487391 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:59.487540 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.488066 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.489055 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.489435 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.489518 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.489526 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.489670 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.489704 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:46:59.489790 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.486992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:35815 (local address 127.11.247.1:40875)
0926 05:46:59.487118 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.487132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.487228 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.487395 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.487396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.487398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:59.487468 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.487470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.488163 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.488164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.488960 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.488962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.489517 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.489518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.489589 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.489593 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.489665 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.489705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260926 05:46:59.489977 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.487137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:33427 (local address 127.11.247.2:35971)
0926 05:46:59.487197 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.487200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.487283 (+ 83us) server_negotiation.cc:407] Connection header received
0926 05:46:59.487378 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.487380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.487390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.487500 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:46:59.487661 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.488137 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.489134 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.489542 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.489659 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.489667 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.489838 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.489885 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:46:59.490105 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.487041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35971 (local address 127.11.247.1:33427)
0926 05:46:59.487188 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.487200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.487300 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.487509 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.487510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.487512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:46:59.487590 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.487591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.488268 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.488269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.489042 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:59.489044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.489668 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.489670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:59.489753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.489756 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.489820 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.490011 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260926 05:46:59.490351 16739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de308913721543a2842b826fe7a7401c" candidate_uuid: "4eacb9199f124c8b91fdb02a3445bac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" is_pre_election: true
I20260926 05:46:59.490511 16739 raft_consensus.cc:2454] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eacb9199f124c8b91fdb02a3445bac6 in term 0.
I20260926 05:46:59.490473 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de308913721543a2842b826fe7a7401c" candidate_uuid: "4eacb9199f124c8b91fdb02a3445bac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" is_pre_election: true
I20260926 05:46:59.490676 16606 raft_consensus.cc:2454] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eacb9199f124c8b91fdb02a3445bac6 in term 0.
I20260926 05:46:59.490926 16406 leader_election.cc:302] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eacb9199f124c8b91fdb02a3445bac6, 589f8dac7e4a410a8c2ec47cfacf56fe; no voters:
I20260926 05:46:59.491073 16809 raft_consensus.cc:2781] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:46:59.491132 16809 raft_consensus.cc:482] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:46:59.491161 16809 raft_consensus.cc:3037] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:59.491837 16809 raft_consensus.cc:504] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.491997 16809 leader_election.cc:288] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [CANDIDATE]: Term 1 election: Requested vote from peers 589f8dac7e4a410a8c2ec47cfacf56fe (127.11.247.3:35815), 0d434ba3524b4e57a1a5e4f42e89e6a2 (127.11.247.2:35971)
I20260926 05:46:59.492153 16739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de308913721543a2842b826fe7a7401c" candidate_uuid: "4eacb9199f124c8b91fdb02a3445bac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe"
I20260926 05:46:59.492254 16739 raft_consensus.cc:3037] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:59.493014 16739 raft_consensus.cc:2454] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eacb9199f124c8b91fdb02a3445bac6 in term 1.
I20260926 05:46:59.493149 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de308913721543a2842b826fe7a7401c" candidate_uuid: "4eacb9199f124c8b91fdb02a3445bac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2"
I20260926 05:46:59.493216 16406 leader_election.cc:302] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eacb9199f124c8b91fdb02a3445bac6, 589f8dac7e4a410a8c2ec47cfacf56fe; no voters:
I20260926 05:46:59.493254 16606 raft_consensus.cc:3037] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:59.493299 16809 raft_consensus.cc:2781] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:59.493541 16809 raft_consensus.cc:686] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [term 1 LEADER]: Becoming Leader. State: Replica: 4eacb9199f124c8b91fdb02a3445bac6, State: Running, Role: LEADER
I20260926 05:46:59.493650 16809 consensus_queue.cc:237] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } }
I20260926 05:46:59.494149 16606 raft_consensus.cc:2454] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eacb9199f124c8b91fdb02a3445bac6 in term 1.
I20260926 05:46:59.494534 16329 catalog_manager.cc:5495] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eacb9199f124c8b91fdb02a3445bac6 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "4eacb9199f124c8b91fdb02a3445bac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eacb9199f124c8b91fdb02a3445bac6" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 45133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 } health_report { overall_health: UNKNOWN } } }
W20260926 05:46:59.520143 16786 tablet.cc:2369] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:46:59.530341 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.527448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.11.247.1:45133)
0926 05:46:59.527557 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.527559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.527568 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:59.527608 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.527609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.527628 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.527693 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:59.527972 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.528469 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.529416 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:59.529619 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:59.529688 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:59.529916 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:59.530033 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:59.530117 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.530213 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.530233 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 05:46:59.532429 16739 raft_consensus.cc:1261] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 1 FOLLOWER]: Refusing update from remote peer 4eacb9199f124c8b91fdb02a3445bac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:59.532454 16606 raft_consensus.cc:1261] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 1 FOLLOWER]: Refusing update from remote peer 4eacb9199f124c8b91fdb02a3445bac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:59.532843 16809 consensus_queue.cc:1035] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589f8dac7e4a410a8c2ec47cfacf56fe" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:59.533020 16813 consensus_queue.cc:1035] T de308913721543a2842b826fe7a7401c P 4eacb9199f124c8b91fdb02a3445bac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d434ba3524b4e57a1a5e4f42e89e6a2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:59.535418 16816 mvcc.cc:205] Tried to move back new op lower bound from 7333485033602277376 to 7333485033447022592. Current Snapshot: MvccSnapshot[applied={T|T < 7333485033602277376}]
I20260926 05:46:59.535905 16818 mvcc.cc:205] Tried to move back new op lower bound from 7333485033602277376 to 7333485033447022592. Current Snapshot: MvccSnapshot[applied={T|T < 7333485033602277376}]
I20260926 05:46:59.538439 16329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:59.538550 16329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:46:59.539410 16329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4454843947b144efb39202e287dda19c: Sending DeleteTablet for 3 replicas of tablet de308913721543a2842b826fe7a7401c
I20260926 05:46:59.539803 16453 tablet_service.cc:1514] Processing DeleteTablet for tablet de308913721543a2842b826fe7a7401c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:59 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60452
I20260926 05:46:59.539855 16719 tablet_service.cc:1514] Processing DeleteTablet for tablet de308913721543a2842b826fe7a7401c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:59 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:34358
I20260926 05:46:59.540053 16586 tablet_service.cc:1514] Processing DeleteTablet for tablet de308913721543a2842b826fe7a7401c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:46:59 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:44412
I20260926 05:46:59.540105 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16391
I20260926 05:46:59.540225 16826 tablet_replica.cc:330] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: stopping tablet replica
I20260926 05:46:59.540870 16827 tablet_replica.cc:330] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: stopping tablet replica
I20260926 05:46:59.541010 16827 raft_consensus.cc:2229] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:59.541126 16827 raft_consensus.cc:2258] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:59.541126 16826 raft_consensus.cc:2229] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:46:59.541281 16826 raft_consensus.cc:2258] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:46:59.541683 16827 ts_tablet_manager.cc:1905] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:59.541803 16826 ts_tablet_manager.cc:1905] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:46:59.543565 16827 ts_tablet_manager.cc:1918] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:46:59.543573 16826 ts_tablet_manager.cc:1918] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:46:59.543648 16827 log.cc:1196] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/de308913721543a2842b826fe7a7401c
I20260926 05:46:59.543648 16826 log.cc:1196] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/de308913721543a2842b826fe7a7401c
I20260926 05:46:59.543937 16826 ts_tablet_manager.cc:1939] T de308913721543a2842b826fe7a7401c P 589f8dac7e4a410a8c2ec47cfacf56fe: Deleting consensus metadata
I20260926 05:46:59.543937 16827 ts_tablet_manager.cc:1939] T de308913721543a2842b826fe7a7401c P 0d434ba3524b4e57a1a5e4f42e89e6a2: Deleting consensus metadata
I20260926 05:46:59.544350 16314 catalog_manager.cc:4841] TS 0d434ba3524b4e57a1a5e4f42e89e6a2 (127.11.247.2:35971): tablet de308913721543a2842b826fe7a7401c (table test-table [id=08ad4d1b2a974ec1aa518ba72197a094]) successfully deleted
I20260926 05:46:59.544386 16313 catalog_manager.cc:4841] TS 589f8dac7e4a410a8c2ec47cfacf56fe (127.11.247.3:35815): tablet de308913721543a2842b826fe7a7401c (table test-table [id=08ad4d1b2a974ec1aa518ba72197a094]) successfully deleted
W20260926 05:46:59.545994 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:45133 (error 108)
I20260926 05:46:59.546291 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16524
I20260926 05:46:59.547096 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.546422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:45133 (local address 127.0.0.1:60468)
0926 05:46:59.546998 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.547050 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547}
W20260926 05:46:59.547260 16314 catalog_manager.cc:4584] TS 4eacb9199f124c8b91fdb02a3445bac6 (127.11.247.1:45133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de308913721543a2842b826fe7a7401c: Network error: Client connection negotiation failed: client connection to 127.11.247.1:45133: connect: Connection refused (error 111)
I20260926 05:46:59.552316 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16657
I20260926 05:46:59.557250 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16298
2026-09-26T05:46:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2139 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T05:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:59Z Disabled control of system clock
I20260926 05:46:59.585806 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34507
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:46:59.667852 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:59.668077 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:59.668099 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:59.669203 16832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:59.669245 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:59.669257 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:59.669270 16832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:59.669282 16832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:59.670758 16832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:59.670967 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:59.671229 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:59.676371 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.676375 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.676375 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.676658 16832 server_base.cc:1034] running on GCE node
I20260926 05:46:59.676901 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:59.677227 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:59.678398 16832 hybrid_clock.cc:648] HybridClock initialized: now 1790401619678383 us; error 60 us; skew 500 ppm
I20260926 05:46:59.679805 16832 webserver.cc:460] Webserver started at http://127.11.247.62:33047/ using document root <none> and password file <none>
I20260926 05:46:59.680119 16832 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:59.680198 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:59.680363 16832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:59.681484 16832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "95e5e29136e543e8a8f0b5dea42033b7"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "d03ad9795b13e72f8909767492fcf709"
server_key_iv: "949c381db184ee358731457ff4a61174"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.681869 16832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "95e5e29136e543e8a8f0b5dea42033b7"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "d03ad9795b13e72f8909767492fcf709"
server_key_iv: "949c381db184ee358731457ff4a61174"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.683350 16832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:59.684201 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.684417 16832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:46:59.684522 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "95e5e29136e543e8a8f0b5dea42033b7"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "d03ad9795b13e72f8909767492fcf709"
server_key_iv: "949c381db184ee358731457ff4a61174"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.684595 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:59.688203 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:59.688506 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:59.688659 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:59.692898 16832 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:34507
I20260926 05:46:59.692942 16898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:34507 every 8 connection(s)
I20260926 05:46:59.693316 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:59.693997 16899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:59.696867 16899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7: Bootstrap starting.
I20260926 05:46:59.697395 16899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:59.697646 16899 log.cc:824] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:59.698283 16899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7: No bootstrap required, opened a new log
I20260926 05:46:59.699903 16899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e5e29136e543e8a8f0b5dea42033b7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34507 } }
I20260926 05:46:59.700042 16899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:46:59.700065 16899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e5e29136e543e8a8f0b5dea42033b7, State: Initialized, Role: FOLLOWER
I20260926 05:46:59.700207 16899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e5e29136e543e8a8f0b5dea42033b7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34507 } }
I20260926 05:46:59.700305 16899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:46:59.700330 16899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:46:59.700398 16899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:59.700501 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16832
I20260926 05:46:59.700613 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:46:59.700781 12252 external_mini_cluster.cc:1428] Setting key fa10f3537139cd05a3235c5eb8d6dd23
I20260926 05:46:59.701157 16899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e5e29136e543e8a8f0b5dea42033b7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34507 } }
I20260926 05:46:59.701277 16899 leader_election.cc:302] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95e5e29136e543e8a8f0b5dea42033b7; no voters:
I20260926 05:46:59.701444 16899 leader_election.cc:288] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:46:59.701524 16904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:59.701674 16904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [term 1 LEADER]: Becoming Leader. State: Replica: 95e5e29136e543e8a8f0b5dea42033b7, State: Running, Role: LEADER
I20260926 05:46:59.701722 16899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:59.701800 16904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e5e29136e543e8a8f0b5dea42033b7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34507 } }
I20260926 05:46:59.702158 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e5e29136e543e8a8f0b5dea42033b7. Latest consensus state: current_term: 1 leader_uuid: "95e5e29136e543e8a8f0b5dea42033b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e5e29136e543e8a8f0b5dea42033b7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34507 } } }
I20260926 05:46:59.702246 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:59.702389 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95e5e29136e543e8a8f0b5dea42033b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e5e29136e543e8a8f0b5dea42033b7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34507 } } }
I20260926 05:46:59.702453 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:59.702559 16912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:59.703125 16912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:59.703753 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.701284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.11.247.62:34507)
0926 05:46:59.701509 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.701518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.701556 (+ 38us) server_negotiation.cc:407] Connection header received
0926 05:46:59.701694 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.701701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.701752 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.702152 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:59.703172 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:59.703176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:59.703176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:46:59.703180 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:59.703214 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:46:59.703227 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:59.703527 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.703570 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.703618 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260926 05:46:59.704793 16912 catalog_manager.cc:1347] Generated new cluster ID: 286d915c6d6b426db0b72cfae69b06ca
I20260926 05:46:59.704844 16912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:46:59.719340 16912 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:59.719547 16912 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:46:59.725136 16912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7: Generated new TSK 0
I20260926 05:46:59.725375 16912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:46:59.731571 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34507
--builtin_ntp_servers=127.11.247.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:46:59.816509 16923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:59.816730 16923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:59.816754 16923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:59.818010 16923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:59.818059 16923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:59.818089 16923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:46:59.819605 16923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:59.819888 16923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:59.820221 16923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:59.825845 16929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.825862 16928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.825914 16931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.826357 16923 server_base.cc:1034] running on GCE node
I20260926 05:46:59.826619 16923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:59.826972 16923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:59.828183 16923 hybrid_clock.cc:648] HybridClock initialized: now 1790401619828155 us; error 34 us; skew 500 ppm
I20260926 05:46:59.829713 16923 webserver.cc:460] Webserver started at http://127.11.247.1:35667/ using document root <none> and password file <none>
I20260926 05:46:59.829991 16923 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:59.830065 16923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:59.830233 16923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:59.831754 16923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "b4c9adc3dc5a1d36bf0b05905b671f83"
server_key_iv: "9f85f35862cceca81a44611a3bf73ad0"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.832206 16923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "b4c9adc3dc5a1d36bf0b05905b671f83"
server_key_iv: "9f85f35862cceca81a44611a3bf73ad0"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.834069 16923 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:46:59.835163 16937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.835364 16923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:46:59.835474 16923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "b4c9adc3dc5a1d36bf0b05905b671f83"
server_key_iv: "9f85f35862cceca81a44611a3bf73ad0"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.835577 16923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:59.839145 16923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:59.839459 16923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:59.839600 16923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:59.839836 16923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:59.840164 16923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:59.840197 16923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.840219 16923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:59.840261 16923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.846331 16923 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:44129
I20260926 05:46:59.846522 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:44129 every 8 connection(s)
I20260926 05:46:59.846874 16923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:46:59.851109 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.847920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:56945 (local address 127.11.247.62:34507)
0926 05:46:59.848002 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.848005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.848345 (+ 340us) server_negotiation.cc:407] Connection header received
0926 05:46:59.848932 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.848934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.848943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.848999 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:59.850810 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:59.850811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:59.850811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:46:59.850818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:59.850845 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:46:59.850849 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:59.850918 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.851004 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.851023 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 05:46:59.851425 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.847922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34507 (local address 127.11.247.1:56945)
0926 05:46:59.848185 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.848235 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.848802 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.849133 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.849137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.849147 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:59.850543 (+ 1396us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:46:59.850543 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:59.850626 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:46:59.850676 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:59.850913 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:59.850921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:59.850925 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.851018 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.851103 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260926 05:46:59.851941 17051 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34507
I20260926 05:46:59.852082 17051 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:59.852279 17051 heartbeater.cc:511] Master 127.11.247.62:34507 requested a full tablet report, sending...
I20260926 05:46:59.852837 16863 ts_manager.cc:194] Registered new tserver with Master: ae81f7b5a7db46d3aaaea9fac81b69e5 (127.11.247.1:44129)
I20260926 05:46:59.856768 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 16923
I20260926 05:46:59.856884 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:46:59.857132 12252 external_mini_cluster.cc:1428] Setting key 9ee387e9f670371c95212fba714d35a9
I20260926 05:46:59.858292 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34507
--builtin_ntp_servers=127.11.247.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:46:59.944096 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:59.944353 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:59.944393 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:59.945658 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:59.945736 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:59.945779 17054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:46:59.947310 17054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:59.947615 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:59.947952 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:59.955581 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.955749 17054 server_base.cc:1034] running on GCE node
W20260926 05:46:59.955581 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:59.955574 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:59.956097 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:59.956332 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:59.957494 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790401619957479 us; error 35 us; skew 500 ppm
I20260926 05:46:59.959017 17054 webserver.cc:460] Webserver started at http://127.11.247.2:37195/ using document root <none> and password file <none>
I20260926 05:46:59.959306 17054 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:59.959358 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:59.959488 17054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:59.960729 17054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "7552fe55d6c74bf495db4626e7193428"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "1366134e8dc47cf2d39ec31ac18b47e1"
server_key_iv: "b94a6e4a0d120c72646a2971e4f41fc9"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.961097 17054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "7552fe55d6c74bf495db4626e7193428"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "1366134e8dc47cf2d39ec31ac18b47e1"
server_key_iv: "b94a6e4a0d120c72646a2971e4f41fc9"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.962543 17054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:59.963461 17068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.963744 17054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:46:59.963827 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "7552fe55d6c74bf495db4626e7193428"
format_stamp: "Formatted at 2026-09-26 05:46:59 on dist-test-slave-21p4"
server_key: "1366134e8dc47cf2d39ec31ac18b47e1"
server_key_iv: "b94a6e4a0d120c72646a2971e4f41fc9"
server_key_version: "encryptionkey@0"
I20260926 05:46:59.963933 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:59.967590 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:59.967876 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:59.968006 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:59.968231 17054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:46:59.968560 17054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:46:59.968592 17054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.968634 17054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:46:59.968660 17054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:59.974828 17054 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:36119
I20260926 05:46:59.974866 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:36119 every 8 connection(s)
I20260926 05:46:59.975245 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:46:59.978681 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.976057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:38123 (local address 127.11.247.62:34507)
0926 05:46:59.976145 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:59.976147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:59.976342 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:46:59.976850 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:59.976850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:59.976856 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:59.976921 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:59.978255 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:59.978256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:59.978256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:46:59.978261 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:59.978277 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:46:59.978281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:59.978335 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:59.978394 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 05:46:59.978540 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:46:59.978695 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:59.976017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34507 (local address 127.11.247.2:38123)
0926 05:46:59.976206 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:59.976249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:59.976740 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:59.977006 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:59.977009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:59.977019 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:59.978079 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:46:59.978080 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:59.978121 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:46:59.978155 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:59.978331 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:59.978334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:59.978336 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:46:59.978391 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:46:59.978445 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260926 05:46:59.979259 17182 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34507
I20260926 05:46:59.979350 17182 heartbeater.cc:464] Registering TS with master...
I20260926 05:46:59.979537 17182 heartbeater.cc:511] Master 127.11.247.62:34507 requested a full tablet report, sending...
I20260926 05:46:59.979938 16863 ts_manager.cc:194] Registered new tserver with Master: 7552fe55d6c74bf495db4626e7193428 (127.11.247.2:36119)
I20260926 05:46:59.983872 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17054
I20260926 05:46:59.983956 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:46:59.984093 12252 external_mini_cluster.cc:1428] Setting key 394c3964a7ee56d8f9b4e930eba16dcb
I20260926 05:46:59.985019 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34507
--builtin_ntp_servers=127.11.247.20:38049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:47:00.082219 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:00.082472 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:00.082499 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:00.083879 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:00.083937 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:00.083967 17185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:00.085328 17185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:00.085567 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:00.085845 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:00.091773 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.091789 17185 server_base.cc:1034] running on GCE node
W20260926 05:47:00.091751 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:00.091754 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.092264 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:00.092562 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:00.093741 17185 hybrid_clock.cc:648] HybridClock initialized: now 1790401620093739 us; error 38 us; skew 500 ppm
I20260926 05:47:00.095296 17185 webserver.cc:460] Webserver started at http://127.11.247.3:34845/ using document root <none> and password file <none>
I20260926 05:47:00.095577 17185 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:00.095655 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:00.095829 17185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:00.097175 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "8b059400a6a748cf9abba9fe68f7ec2e"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "6e8cdfa595d175f29ff4c2b1ede57393"
server_key_iv: "6482d41b1d8513a1322fae79d02fc5f2"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.097581 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "8b059400a6a748cf9abba9fe68f7ec2e"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "6e8cdfa595d175f29ff4c2b1ede57393"
server_key_iv: "6482d41b1d8513a1322fae79d02fc5f2"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.099431 17185 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 05:47:00.100435 17199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:00.100693 17185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:00.100783 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "8b059400a6a748cf9abba9fe68f7ec2e"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "6e8cdfa595d175f29ff4c2b1ede57393"
server_key_iv: "6482d41b1d8513a1322fae79d02fc5f2"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.100841 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:00.104669 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:00.104918 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:00.105012 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:00.105180 17185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:00.105525 17185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:00.105566 17185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:00.105587 17185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:00.105600 17185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:00.111351 17185 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:43389
I20260926 05:47:00.111410 17312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:43389 every 8 connection(s)
I20260926 05:47:00.111796 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:00.116083 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.112805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:57951 (local address 127.11.247.62:34507)
0926 05:47:00.112941 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.112944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.113193 (+ 249us) server_negotiation.cc:407] Connection header received
0926 05:47:00.113966 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.113967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.113974 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.114233 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.115795 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.115796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.115796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.115801 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.115819 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.115826 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.115899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.115982 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.116001 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 05:47:00.116340 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.112751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34507 (local address 127.11.247.3:57951)
0926 05:47:00.113063 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.113100 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.113626 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.114318 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.114323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.114340 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:00.115601 (+ 1261us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:00.115602 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:00.115635 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:00.115668 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:00.115897 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:00.115903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:00.115906 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:00.115994 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:47:00.116074 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":116,"threads_started":1}
I20260926 05:47:00.116988 17313 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34507
I20260926 05:47:00.117129 17313 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:00.117321 17313 heartbeater.cc:511] Master 127.11.247.62:34507 requested a full tablet report, sending...
I20260926 05:47:00.117762 16863 ts_manager.cc:194] Registered new tserver with Master: 8b059400a6a748cf9abba9fe68f7ec2e (127.11.247.3:43389)
I20260926 05:47:00.121088 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17185
I20260926 05:47:00.121191 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:00.121385 12252 external_mini_cluster.cc:1428] Setting key 44a6f58fbffb5fd8b5dee89bc7cf59b9
I20260926 05:47:00.122428 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:47:00.124941 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.123902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.11.247.62:34507)
0926 05:47:00.123973 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.123975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.124068 (+ 93us) server_negotiation.cc:407] Connection header received
0926 05:47:00.124144 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.124144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.124150 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.124191 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.124756 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.124756 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.124757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.124761 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.124777 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.124781 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.124825 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.124858 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.124872 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:47:00.131263 16863 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:47:00.133530 16863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:00.138890 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.136820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:36119 (local address 127.0.0.1:39284)
0926 05:47:00.137040 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.137087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.137153 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.137578 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.137585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.137593 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:00.138316 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:00.138317 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:00.138337 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:00.138365 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:00.138698 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:00.138706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:00.138710 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:00.138782 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:47:00.138799 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260926 05:47:00.139084 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.136820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:43389 (local address 127.0.0.1:58730)
0926 05:47:00.137234 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.137247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.137288 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.137620 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.137621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.137624 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:00.138646 (+ 1022us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:00.138647 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:00.138677 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:00.138693 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:00.138962 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:00.138963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:00.138964 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:47:00.138996 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 05:47:00.139008 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":4,"thread_start_us":118,"threads_started":1}
I20260926 05:47:00.139187 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.137285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.11.247.1:44129)
0926 05:47:00.137555 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.137559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.137727 (+ 168us) server_negotiation.cc:407] Connection header received
0926 05:47:00.137759 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.137761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.137772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.137836 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.138762 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.138769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.138769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.138774 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.138806 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.138857 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.138939 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.139052 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.139073 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":91,"threads_started":1}
I20260926 05:47:00.139390 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.137072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.11.247.3:43389)
0926 05:47:00.137428 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.137433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.137444 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:00.137471 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.137474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.137501 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.138132 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.138803 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.138809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.138810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.138814 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.138847 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.138871 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.139242 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.139269 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.139285 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":100,"threads_started":1}
I20260926 05:47:00.139479 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.136828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:44129 (local address 127.0.0.1:56336)
0926 05:47:00.137622 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.137631 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.137673 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.138172 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.138173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.138176 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:00.138653 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:00.138654 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:00.138662 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:00.138677 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:00.138967 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:00.138968 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:00.138969 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:47:00.139380 (+ 411us) client_negotiation.cc:240] Negotiation successful
0926 05:47:00.139395 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":23,"thread_start_us":181,"threads_started":1}
I20260926 05:47:00.139635 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.137175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.11.247.2:36119)
0926 05:47:00.137390 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.137396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.137408 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:47:00.137438 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.137441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.137452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.137577 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.138467 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.138476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.138477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.138481 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.138550 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.138619 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.138697 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.139536 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.139553 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260926 05:47:00.140220 16985 tablet_service.cc:1467] Processing CreateTablet for tablet 0584f3b112424ba58cb3d051cd40beb0 (DEFAULT_TABLE table=test-table [id=c9c7c779af4f422da5d732345049c366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:00.140219 17247 tablet_service.cc:1467] Processing CreateTablet for tablet 0584f3b112424ba58cb3d051cd40beb0 (DEFAULT_TABLE table=test-table [id=c9c7c779af4f422da5d732345049c366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:00.140301 17116 tablet_service.cc:1467] Processing CreateTablet for tablet 0584f3b112424ba58cb3d051cd40beb0 (DEFAULT_TABLE table=test-table [id=c9c7c779af4f422da5d732345049c366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:00.140625 17247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0584f3b112424ba58cb3d051cd40beb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:00.140625 16985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0584f3b112424ba58cb3d051cd40beb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:00.140647 17116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0584f3b112424ba58cb3d051cd40beb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:00.143780 17327 tablet_bootstrap.cc:492] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: Bootstrap starting.
I20260926 05:47:00.144549 17327 tablet_bootstrap.cc:654] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:00.144898 17329 tablet_bootstrap.cc:492] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: Bootstrap starting.
I20260926 05:47:00.144941 17327 log.cc:824] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:00.144990 17328 tablet_bootstrap.cc:492] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Bootstrap starting.
I20260926 05:47:00.145617 17329 tablet_bootstrap.cc:654] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:00.145608 17328 tablet_bootstrap.cc:654] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:00.145787 17327 tablet_bootstrap.cc:492] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: No bootstrap required, opened a new log
I20260926 05:47:00.145880 17327 ts_tablet_manager.cc:1397] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:47:00.145938 17329 log.cc:824] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:00.146039 17328 log.cc:824] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:00.146823 17329 tablet_bootstrap.cc:492] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: No bootstrap required, opened a new log
I20260926 05:47:00.146895 17329 ts_tablet_manager.cc:1397] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:00.146981 17328 tablet_bootstrap.cc:492] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: No bootstrap required, opened a new log
I20260926 05:47:00.147045 17328 ts_tablet_manager.cc:1397] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:47:00.148803 17327 raft_consensus.cc:348] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.149076 17327 raft_consensus.cc:374] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:00.149137 17327 raft_consensus.cc:729] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7552fe55d6c74bf495db4626e7193428, State: Initialized, Role: FOLLOWER
I20260926 05:47:00.149302 17327 consensus_queue.cc:260] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.149672 17327 ts_tablet_manager.cc:1428] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 05:47:00.149844 17182 heartbeater.cc:503] Master 127.11.247.62:34507 was elected leader, sending a full tablet report...
I20260926 05:47:00.149801 17329 raft_consensus.cc:348] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.149951 17329 raft_consensus.cc:374] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:00.149999 17329 raft_consensus.cc:729] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae81f7b5a7db46d3aaaea9fac81b69e5, State: Initialized, Role: FOLLOWER
I20260926 05:47:00.149688 17328 raft_consensus.cc:348] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.150128 17329 consensus_queue.cc:260] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.150209 17328 raft_consensus.cc:374] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:00.150256 17328 raft_consensus.cc:729] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b059400a6a748cf9abba9fe68f7ec2e, State: Initialized, Role: FOLLOWER
I20260926 05:47:00.150404 17329 ts_tablet_manager.cc:1428] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:47:00.150390 17328 consensus_queue.cc:260] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.150524 17051 heartbeater.cc:503] Master 127.11.247.62:34507 was elected leader, sending a full tablet report...
I20260926 05:47:00.150966 17328 ts_tablet_manager.cc:1428] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:47:00.151090 17313 heartbeater.cc:503] Master 127.11.247.62:34507 was elected leader, sending a full tablet report...
I20260926 05:47:00.154181 17334 raft_consensus.cc:482] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:00.154302 17334 raft_consensus.cc:504] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.154886 17334 leader_election.cc:288] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7552fe55d6c74bf495db4626e7193428 (127.11.247.2:36119), 8b059400a6a748cf9abba9fe68f7ec2e (127.11.247.3:43389)
I20260926 05:47:00.156553 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.155132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:36119 (local address 127.11.247.1:54681)
0926 05:47:00.155206 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.155220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.155261 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.155464 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.155465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.155468 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:00.155911 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:00.155912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:00.155922 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:00.155938 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:00.156417 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:00.156418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:00.156419 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:47:00.156485 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:47:00.156499 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 05:47:00.156566 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.155223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:54681 (local address 127.11.247.2:36119)
0926 05:47:00.155272 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.155274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.155285 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:00.155319 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.155320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.155327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.155399 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.156111 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.156112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.156113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.156119 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.156138 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.156143 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.156204 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.156494 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.156511 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:47:00.156909 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.155383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:47327 (local address 127.11.247.3:43389)
0926 05:47:00.155427 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.155429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.155438 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:00.155465 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.155465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.155470 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.155529 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.156682 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.156683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.156684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.156689 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.156708 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.156713 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.156776 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.156829 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.156845 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:47:00.156922 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.155166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:43389 (local address 127.11.247.1:47327)
0926 05:47:00.155343 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.155355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.155402 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.155531 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.155531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.155533 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:00.156123 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:00.156124 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:00.156130 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:00.156139 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:00.156781 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:00.156782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:00.156783 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:47:00.156837 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 05:47:00.156849 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":120,"threads_started":1}
I20260926 05:47:00.156898 17136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0584f3b112424ba58cb3d051cd40beb0" candidate_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7552fe55d6c74bf495db4626e7193428" is_pre_election: true
I20260926 05:47:00.157116 17136 raft_consensus.cc:2454] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae81f7b5a7db46d3aaaea9fac81b69e5 in term 0.
I20260926 05:47:00.157276 17267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0584f3b112424ba58cb3d051cd40beb0" candidate_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" is_pre_election: true
I20260926 05:47:00.157430 17267 raft_consensus.cc:2454] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae81f7b5a7db46d3aaaea9fac81b69e5 in term 0.
I20260926 05:47:00.157428 16938 leader_election.cc:302] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7552fe55d6c74bf495db4626e7193428, ae81f7b5a7db46d3aaaea9fac81b69e5; no voters:
I20260926 05:47:00.157570 17334 raft_consensus.cc:2781] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:00.157629 17334 raft_consensus.cc:482] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:00.157665 17334 raft_consensus.cc:3037] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:00.188905 17334 raft_consensus.cc:504] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.189175 17334 leader_election.cc:288] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [CANDIDATE]: Term 1 election: Requested vote from peers 7552fe55d6c74bf495db4626e7193428 (127.11.247.2:36119), 8b059400a6a748cf9abba9fe68f7ec2e (127.11.247.3:43389)
I20260926 05:47:00.189384 17136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0584f3b112424ba58cb3d051cd40beb0" candidate_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7552fe55d6c74bf495db4626e7193428"
I20260926 05:47:00.189499 17267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0584f3b112424ba58cb3d051cd40beb0" candidate_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b059400a6a748cf9abba9fe68f7ec2e"
I20260926 05:47:00.189533 17136 raft_consensus.cc:3037] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:00.189620 17267 raft_consensus.cc:3037] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:00.190549 17267 raft_consensus.cc:2454] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae81f7b5a7db46d3aaaea9fac81b69e5 in term 1.
I20260926 05:47:00.190544 17136 raft_consensus.cc:2454] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae81f7b5a7db46d3aaaea9fac81b69e5 in term 1.
I20260926 05:47:00.190820 16941 leader_election.cc:302] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b059400a6a748cf9abba9fe68f7ec2e, ae81f7b5a7db46d3aaaea9fac81b69e5; no voters:
I20260926 05:47:00.190980 17334 raft_consensus.cc:2781] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:00.191177 17334 raft_consensus.cc:686] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 1 LEADER]: Becoming Leader. State: Replica: ae81f7b5a7db46d3aaaea9fac81b69e5, State: Running, Role: LEADER
I20260926 05:47:00.191299 17334 consensus_queue.cc:237] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } }
I20260926 05:47:00.192261 16862 catalog_manager.cc:5495] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae81f7b5a7db46d3aaaea9fac81b69e5 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae81f7b5a7db46d3aaaea9fac81b69e5" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44129 } health_report { overall_health: HEALTHY } } }
I20260926 05:47:00.201855 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.200707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.11.247.1:44129)
0926 05:47:00.200770 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.200772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.200781 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:00.200821 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.200822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.200837 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.200887 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.201625 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.201626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.201626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.201630 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.201643 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.201647 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.201695 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.201725 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.201737 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:47:00.204244 17136 raft_consensus.cc:1261] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 1 FOLLOWER]: Refusing update from remote peer ae81f7b5a7db46d3aaaea9fac81b69e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:00.204281 17267 raft_consensus.cc:1261] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 1 FOLLOWER]: Refusing update from remote peer ae81f7b5a7db46d3aaaea9fac81b69e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:00.204646 17337 consensus_queue.cc:1035] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b059400a6a748cf9abba9fe68f7ec2e" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:00.204778 17334 consensus_queue.cc:1035] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7552fe55d6c74bf495db4626e7193428" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:00.208554 17342 mvcc.cc:205] Tried to move back new op lower bound from 7333485036353343488 to 7333485036304367616. Current Snapshot: MvccSnapshot[applied={T|T < 7333485036353343488 or (T in {7333485036353343488})}]
I20260926 05:47:00.208734 17341 mvcc.cc:205] Tried to move back new op lower bound from 7333485036353343488 to 7333485036304367616. Current Snapshot: MvccSnapshot[applied={T|T < 7333485036353343488}]
I20260926 05:47:00.210023 16863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:00.210136 16863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:00.211053 16863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95e5e29136e543e8a8f0b5dea42033b7: Sending DeleteTablet for 3 replicas of tablet 0584f3b112424ba58cb3d051cd40beb0
I20260926 05:47:00.211439 17247 tablet_service.cc:1514] Processing DeleteTablet for tablet 0584f3b112424ba58cb3d051cd40beb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:00 UTC) from {username='slave'} at 127.0.0.1:58730
I20260926 05:47:00.211440 16985 tablet_service.cc:1514] Processing DeleteTablet for tablet 0584f3b112424ba58cb3d051cd40beb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:00 UTC) from {username='slave'} at 127.0.0.1:56336
I20260926 05:47:00.211458 17116 tablet_service.cc:1514] Processing DeleteTablet for tablet 0584f3b112424ba58cb3d051cd40beb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:00 UTC) from {username='slave'} at 127.0.0.1:39284
I20260926 05:47:00.211956 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16923
I20260926 05:47:00.212013 17351 tablet_replica.cc:330] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5: stopping tablet replica
I20260926 05:47:00.212106 17351 raft_consensus.cc:2229] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:00.212188 17351 raft_consensus.cc:2258] T 0584f3b112424ba58cb3d051cd40beb0 P ae81f7b5a7db46d3aaaea9fac81b69e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:00.212591 17350 tablet_replica.cc:330] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: stopping tablet replica
I20260926 05:47:00.212682 17350 raft_consensus.cc:2229] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:00.212760 17350 raft_consensus.cc:2258] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:00.212929 17352 tablet_replica.cc:330] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: stopping tablet replica
I20260926 05:47:00.213027 17352 raft_consensus.cc:2229] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:00.213096 17352 raft_consensus.cc:2258] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:00.213301 17350 ts_tablet_manager.cc:1905] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:00.213506 17352 ts_tablet_manager.cc:1905] T 0584f3b112424ba58cb3d051cd40beb0 P 7552fe55d6c74bf495db4626e7193428: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:47:00.218883 16849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:44129 (error 108)
I20260926 05:47:00.219558 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.219384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:44129 (local address 127.0.0.1:56356)
0926 05:47:00.219459 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.219511 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:44129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:47:00.219754 16849 catalog_manager.cc:4584] TS ae81f7b5a7db46d3aaaea9fac81b69e5 (127.11.247.1:44129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0584f3b112424ba58cb3d051cd40beb0: Network error: Client connection negotiation failed: client connection to 127.11.247.1:44129: connect: Connection refused (error 111)
I20260926 05:47:00.219931 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17054
I20260926 05:47:00.272558 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.272342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:44129 (local address 127.0.0.1:53656)
0926 05:47:00.272454 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.272506 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:44129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 05:47:00.273173 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.273046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:44129 (local address 127.0.0.1:53672)
0926 05:47:00.273110 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.273148 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:44129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:47:00.278702 17350 ts_tablet_manager.cc:1918] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:00.278827 17350 log.cc:1196] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/0584f3b112424ba58cb3d051cd40beb0
I20260926 05:47:00.279145 17350 ts_tablet_manager.cc:1939] T 0584f3b112424ba58cb3d051cd40beb0 P 8b059400a6a748cf9abba9fe68f7ec2e: Deleting consensus metadata
I20260926 05:47:00.281495 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17185
I20260926 05:47:00.281587 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.281413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:36119 (local address 127.0.0.1:60960)
0926 05:47:00.281492 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.281527 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.2:36119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 05:47:00.304105 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 16832
I20260926 05:47:00.304195 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.304010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:43389 (local address 127.0.0.1:55704)
0926 05:47:00.304096 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.304133 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.3:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
2026-09-26T05:47:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (746 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 05:47:00.324877 12252 mini_oidc.cc:152] Starting JWKS server
I20260926 05:47:00.325976 12252 webserver.cc:460] Webserver started at https://127.0.0.1:46793/ using document root <none> and password file <none>
I20260926 05:47:00.326112 12252 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:47:00.326269 12252 webserver.cc:460] Webserver started at http://127.0.0.1:45827/ using document root <none> and password file <none>
2026-09-26T05:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:00Z Disabled control of system clock
I20260926 05:47:00.335296 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42909
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42909
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46793/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/testchainca.cert with env {}
W20260926 05:47:00.427011 17359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:00.427246 17359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:00.427271 17359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:00.428571 17359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:00.428624 17359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:00.428637 17359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:00.428650 17359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:00.428663 17359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:47:00.428675 17359 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:47:00.428687 17359 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46793/jwks.json
I20260926 05:47:00.430425 17359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46793/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:00.430773 17359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:00.430975 17359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:00.436218 17364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:00.436184 17367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.436486 17359 server_base.cc:1034] running on GCE node
W20260926 05:47:00.436197 17365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.436841 17359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:00.437165 17359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:00.438342 17359 hybrid_clock.cc:648] HybridClock initialized: now 1790401620438324 us; error 31 us; skew 500 ppm
I20260926 05:47:00.439860 17359 webserver.cc:460] Webserver started at http://127.11.247.62:37783/ using document root <none> and password file <none>
I20260926 05:47:00.440126 17359 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:00.440178 17359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:00.440317 17359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:00.441459 17359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "b63d497f72a0449781b9cc678df85c59"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "87f4911f0558754f8ada151385964513"
server_key_iv: "e7a2346b8cfa6e6a3c8a1dad0c2ccf57"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.441845 17359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "b63d497f72a0449781b9cc678df85c59"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "87f4911f0558754f8ada151385964513"
server_key_iv: "e7a2346b8cfa6e6a3c8a1dad0c2ccf57"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.443344 17359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:47:00.444136 17373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:00.444340 17359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:00.444478 17359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "b63d497f72a0449781b9cc678df85c59"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "87f4911f0558754f8ada151385964513"
server_key_iv: "e7a2346b8cfa6e6a3c8a1dad0c2ccf57"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.444561 17359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:00.453244 17359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:00.453601 17359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:00.453764 17359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:00.458185 17359 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:42909
I20260926 05:47:00.458249 17425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:42909 every 8 connection(s)
I20260926 05:47:00.458709 17359 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:00.459757 17426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:00.460099 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17359
I20260926 05:47:00.460206 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:00.460371 12252 external_mini_cluster.cc:1428] Setting key addebb352f725f65a0f03f39afbc6f39
I20260926 05:47:00.462411 17426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59: Bootstrap starting.
I20260926 05:47:00.463135 17426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:00.463449 17426 log.cc:824] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:00.464279 17426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59: No bootstrap required, opened a new log
I20260926 05:47:00.467201 17426 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63d497f72a0449781b9cc678df85c59" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42909 } }
I20260926 05:47:00.467338 17426 raft_consensus.cc:374] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:00.467361 17426 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63d497f72a0449781b9cc678df85c59, State: Initialized, Role: FOLLOWER
I20260926 05:47:00.467501 17426 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63d497f72a0449781b9cc678df85c59" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42909 } }
I20260926 05:47:00.467572 17426 raft_consensus.cc:388] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:00.467612 17426 raft_consensus.cc:482] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:00.467665 17426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:00.470022 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.464027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.11.247.62:42909)
0926 05:47:00.464236 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.464246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.464278 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:47:00.464876 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.464886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.464969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.465131 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.465360 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.466941 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.467953 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.468556 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.468652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:00.469306 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.469310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.469311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.469528 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.469574 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.469586 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.469642 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.469714 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.469806 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260926 05:47:00.471127 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:47:00.556304 17426 raft_consensus.cc:504] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63d497f72a0449781b9cc678df85c59" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42909 } }
I20260926 05:47:00.556500 17426 leader_election.cc:302] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b63d497f72a0449781b9cc678df85c59; no voters:
I20260926 05:47:00.556754 17426 leader_election.cc:288] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:00.556842 17431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:00.557026 17431 raft_consensus.cc:686] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [term 1 LEADER]: Becoming Leader. State: Replica: b63d497f72a0449781b9cc678df85c59, State: Running, Role: LEADER
I20260926 05:47:00.557107 17426 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:00.557161 17431 consensus_queue.cc:237] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63d497f72a0449781b9cc678df85c59" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42909 } }
I20260926 05:47:00.557814 17432 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b63d497f72a0449781b9cc678df85c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63d497f72a0449781b9cc678df85c59" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42909 } } }
I20260926 05:47:00.557912 17432 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:00.557940 17433 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63d497f72a0449781b9cc678df85c59. Latest consensus state: current_term: 1 leader_uuid: "b63d497f72a0449781b9cc678df85c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63d497f72a0449781b9cc678df85c59" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42909 } } }
I20260926 05:47:00.558127 17433 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:00.558431 17437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:00.559090 17437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:00.560981 17437 catalog_manager.cc:1347] Generated new cluster ID: 8ed31c6e1f03427eb7082787defe4921
I20260926 05:47:00.561039 17437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:00.567804 17437 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:00.568564 17437 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:00.576790 17437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b63d497f72a0449781b9cc678df85c59: Generated new TSK 0
I20260926 05:47:00.577214 17437 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 05:47:00.614955 17457 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 05:47:00.615078 17457 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 05:47:00.615331 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.599686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.11.247.62:42909)
0926 05:47:00.599739 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.599742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.599884 (+ 142us) server_negotiation.cc:407] Connection header received
0926 05:47:00.600003 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.600003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.600016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.600080 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:47:00.600257 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.600786 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.601960 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.602163 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.602217 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:00.602480 (+ 263us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:47:00.615087 (+ 12607us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46793/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 05:47:00.615262 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33088: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46793/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 05:47:00.615415 17430 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33088: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46793/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 05:47:00.615716 17456 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:00.599556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42909 (local address 127.0.0.1:33088)
0926 05:47:00.599797 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.599809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.599890 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.600111 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.600112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.600114 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:47:00.600187 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:00.600190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.600864 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.600865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:00.601866 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:00.601870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.602248 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.602249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:00.602338 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:00.602364 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:47:00.615282 (+ 12918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46793/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 05:47:00.615396 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.247.62:42909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46793/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":136,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.247.62:42909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46793/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 05:47:00.616158 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17359
2026-09-26T05:47:00Z chronyd exiting
I20260926 05:47:00.652518 12252 test_util.cc:175] -----------------------------------------------
I20260926 05:47:00.652628 12252 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401595249435-12252-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 05:47:00.653431 12252 mini_oidc.cc:152] Starting JWKS server
I20260926 05:47:00.654166 12252 webserver.cc:460] Webserver started at https://127.0.0.1:36827/ using document root <none> and password file <none>
I20260926 05:47:00.654239 12252 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:47:00.654428 12252 webserver.cc:460] Webserver started at http://127.0.0.1:44089/ using document root <none> and password file <none>
2026-09-26T05:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:00Z Disabled control of system clock
I20260926 05:47:00.662823 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:44193
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:44193
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36827/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/testchainca.cert with env {}
W20260926 05:47:00.748854 17463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:00.749079 17463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:00.749104 17463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:00.750258 17463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:00.750304 17463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:00.750319 17463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:00.750331 17463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:00.750348 17463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:47:00.750361 17463 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:47:00.750373 17463 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36827/jwks.json
I20260926 05:47:00.751919 17463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:44193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36827/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:00.752128 17463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:00.752305 17463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:00.757537 17471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.757568 17463 server_base.cc:1034] running on GCE node
W20260926 05:47:00.757537 17468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:00.757578 17469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.758003 17463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:00.758289 17463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:00.759445 17463 hybrid_clock.cc:648] HybridClock initialized: now 1790401620759410 us; error 51 us; skew 500 ppm
I20260926 05:47:00.760805 17463 webserver.cc:460] Webserver started at http://127.11.247.62:44451/ using document root <none> and password file <none>
I20260926 05:47:00.761098 17463 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:00.761179 17463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:00.761332 17463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:00.762454 17463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "3510e6f672424d0896f7abc10305928e"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "f81e1ea307c0d052da135ce86435aef0"
server_key_iv: "25f634e935b186ad1c074ac101d733b2"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.762892 17463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "3510e6f672424d0896f7abc10305928e"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "f81e1ea307c0d052da135ce86435aef0"
server_key_iv: "25f634e935b186ad1c074ac101d733b2"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.764329 17463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:00.765149 17477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:00.765378 17463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:00.765497 17463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "3510e6f672424d0896f7abc10305928e"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "f81e1ea307c0d052da135ce86435aef0"
server_key_iv: "25f634e935b186ad1c074ac101d733b2"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.765578 17463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:00.775718 17463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:00.776113 17463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:00.776288 17463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:00.780864 17463 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:44193
I20260926 05:47:00.780901 17529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:44193 every 8 connection(s)
I20260926 05:47:00.781276 17463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:00.782020 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:00.785020 17530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e: Bootstrap starting.
I20260926 05:47:00.785688 17530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:00.786046 17530 log.cc:824] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:00.786916 17530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e: No bootstrap required, opened a new log
I20260926 05:47:00.787488 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17463
I20260926 05:47:00.787631 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:00.787832 12252 external_mini_cluster.cc:1428] Setting key d23434892deafa78f03976c24e1f84da
I20260926 05:47:00.789105 17530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3510e6f672424d0896f7abc10305928e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44193 } }
I20260926 05:47:00.789284 17530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:00.789335 17530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3510e6f672424d0896f7abc10305928e, State: Initialized, Role: FOLLOWER
I20260926 05:47:00.789515 17530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3510e6f672424d0896f7abc10305928e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44193 } }
I20260926 05:47:00.789623 17530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:00.789664 17530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:00.789718 17530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:00.790333 17530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3510e6f672424d0896f7abc10305928e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44193 } }
I20260926 05:47:00.790457 17530 leader_election.cc:302] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3510e6f672424d0896f7abc10305928e; no voters:
I20260926 05:47:00.790696 17530 leader_election.cc:288] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:00.790740 17535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:00.790886 17535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [term 1 LEADER]: Becoming Leader. State: Replica: 3510e6f672424d0896f7abc10305928e, State: Running, Role: LEADER
I20260926 05:47:00.790977 17535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3510e6f672424d0896f7abc10305928e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44193 } }
I20260926 05:47:00.790985 17530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:00.791354 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3510e6f672424d0896f7abc10305928e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3510e6f672424d0896f7abc10305928e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44193 } } }
I20260926 05:47:00.791419 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:00.791549 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3510e6f672424d0896f7abc10305928e. Latest consensus state: current_term: 1 leader_uuid: "3510e6f672424d0896f7abc10305928e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3510e6f672424d0896f7abc10305928e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44193 } } }
I20260926 05:47:00.791621 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e [sys.catalog]: This master's current role is: LEADER
W20260926 05:47:00.793087 17551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:47:00.793149 17551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:47:00.793198 17540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:00.793345 17540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:00.795014 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.788424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.11.247.62:44193)
0926 05:47:00.788649 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.788658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.788694 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:47:00.789100 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.789109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.789209 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.789402 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:00.789696 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.791524 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.792681 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.793136 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.793650 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:00.794052 (+ 402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:00.794056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:00.794057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:00.794265 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:00.794306 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:00.794321 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:00.794389 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:00.794502 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:47:00.794629 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260926 05:47:00.795696 17540 catalog_manager.cc:1347] Generated new cluster ID: ae5dd14da3534eb6bb322eea13ea01de
I20260926 05:47:00.795754 17540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:00.796060 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:47:00.803892 17540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:00.804625 17540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:00.807909 17540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3510e6f672424d0896f7abc10305928e: Generated new TSK 0
I20260926 05:47:00.808111 17540 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 05:47:00.828662 17560 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 05:47:00.828774 17560 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 05:47:00.829087 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:00.813384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.11.247.62:44193)
0926 05:47:00.813445 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:00.813448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:00.813491 (+ 43us) server_negotiation.cc:407] Connection header received
0926 05:47:00.813570 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:00.813571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:00.813584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:00.813674 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:47:00.813816 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.814647 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.815535 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.815856 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.815933 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:00.816171 (+ 238us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:47:00.828851 (+ 12680us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 05:47:00.829012 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 05:47:00.829149 17534 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 05:47:00.829221 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:00.813273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:44193 (local address 127.0.0.1:46410)
0926 05:47:00.813419 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:00.813431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:00.813501 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:00.813680 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:00.813681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:00.813683 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:47:00.813740 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:00.813743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.814796 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.814797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:00.815426 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:00.815429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:00.815979 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:00.815980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:00.816035 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:00.816041 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:47:00.828984 (+ 12943us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 05:47:00.829099 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.247.62:44193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 05:47:00.829598 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17463
2026-09-26T05:47:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 05:47:00.867740 12252 mini_oidc.cc:152] Starting JWKS server
I20260926 05:47:00.868422 12252 webserver.cc:460] Webserver started at https://127.0.0.1:34217/ using document root <none> and password file <none>
I20260926 05:47:00.868465 12252 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:47:00.868603 12252 webserver.cc:460] Webserver started at http://127.0.0.1:45309/ using document root <none> and password file <none>
2026-09-26T05:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:00Z Disabled control of system clock
I20260926 05:47:00.876993 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:35327
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:35327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34217/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 05:47:00.961889 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:00.962216 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:00.962260 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:00.963577 17566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:00.963666 17566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:00.963690 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:00.963722 17566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:00.963749 17566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:47:00.963773 17566 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:47:00.963800 17566 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34217/jwks.json
I20260926 05:47:00.965260 17566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:35327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34217/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:00.965580 17566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:00.965919 17566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:47:00.971382 17566 server_base.cc:1034] running on GCE node
W20260926 05:47:00.971470 17574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:00.971351 17571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:00.971349 17572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:00.971985 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:00.972273 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:00.973460 17566 hybrid_clock.cc:648] HybridClock initialized: now 1790401620973453 us; error 55 us; skew 500 ppm
I20260926 05:47:00.974965 17566 webserver.cc:460] Webserver started at http://127.11.247.62:40589/ using document root <none> and password file <none>
I20260926 05:47:00.975265 17566 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:00.975340 17566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:00.975512 17566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:00.976666 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "9d8fb56367c9449599be5f33e1a116c7"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "293b51005a83d8e1ac290d583b16613c"
server_key_iv: "4e6257d45c682cae0abd066d75819118"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.977064 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "9d8fb56367c9449599be5f33e1a116c7"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "293b51005a83d8e1ac290d583b16613c"
server_key_iv: "4e6257d45c682cae0abd066d75819118"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.978503 17566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:47:00.979467 17580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:00.979708 17566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:00.979838 17566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "9d8fb56367c9449599be5f33e1a116c7"
format_stamp: "Formatted at 2026-09-26 05:47:00 on dist-test-slave-21p4"
server_key: "293b51005a83d8e1ac290d583b16613c"
server_key_iv: "4e6257d45c682cae0abd066d75819118"
server_key_version: "encryptionkey@0"
I20260926 05:47:00.979930 17566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:00.992661 17566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:00.992991 17566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:00.993146 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:00.997300 17566 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:35327
I20260926 05:47:00.997354 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:35327 every 8 connection(s)
I20260926 05:47:00.997756 17566 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:00.998490 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:01.001503 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7: Bootstrap starting.
I20260926 05:47:01.002161 17633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:01.002457 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17566
I20260926 05:47:01.002513 17633 log.cc:824] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:01.002565 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:01.002780 12252 external_mini_cluster.cc:1428] Setting key 03117b2a70a9f2cb86032772113c4b16
I20260926 05:47:01.003545 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7: No bootstrap required, opened a new log
I20260926 05:47:01.006358 17633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8fb56367c9449599be5f33e1a116c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35327 } }
I20260926 05:47:01.006605 17633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:01.006650 17633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8fb56367c9449599be5f33e1a116c7, State: Initialized, Role: FOLLOWER
I20260926 05:47:01.006825 17633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8fb56367c9449599be5f33e1a116c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35327 } }
I20260926 05:47:01.006898 17633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:01.006923 17633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:01.006999 17633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:01.007826 17633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8fb56367c9449599be5f33e1a116c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35327 } }
I20260926 05:47:01.007956 17633 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8fb56367c9449599be5f33e1a116c7; no voters:
I20260926 05:47:01.008157 17633 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:01.008275 17638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:01.008440 17638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [term 1 LEADER]: Becoming Leader. State: Replica: 9d8fb56367c9449599be5f33e1a116c7, State: Running, Role: LEADER
I20260926 05:47:01.008472 17633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:01.008553 17638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8fb56367c9449599be5f33e1a116c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35327 } }
I20260926 05:47:01.008893 17639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8fb56367c9449599be5f33e1a116c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8fb56367c9449599be5f33e1a116c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35327 } } }
I20260926 05:47:01.009006 17639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:01.009235 17640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8fb56367c9449599be5f33e1a116c7. Latest consensus state: current_term: 1 leader_uuid: "9d8fb56367c9449599be5f33e1a116c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8fb56367c9449599be5f33e1a116c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35327 } } }
I20260926 05:47:01.009300 17640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:01.009323 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:01.003341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.11.247.62:35327)
0926 05:47:01.003561 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:01.003569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:01.003595 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:47:01.004129 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:01.004136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:01.004205 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:01.004330 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:01.004655 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.005552 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.006750 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.007359 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.007470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:01.008287 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:01.008293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:01.008293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:01.008507 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:01.008554 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:01.008570 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:01.008636 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:01.008944 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 05:47:01.009053 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260926 05:47:01.009421 17645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:01.009963 17645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:01.010625 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:47:01.012353 17645 catalog_manager.cc:1347] Generated new cluster ID: 594631f60fe3462b85b297ff8503b403
I20260926 05:47:01.012429 17645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:01.019001 17645 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:01.019564 17645 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:01.028693 17645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8fb56367c9449599be5f33e1a116c7: Generated new TSK 0
I20260926 05:47:01.028923 17645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:47:01.046000 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:01.042477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.11.247.62:35327)
0926 05:47:01.042532 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:01.042534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:01.042639 (+ 105us) server_negotiation.cc:407] Connection header received
0926 05:47:01.042730 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:01.042731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:01.042745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:01.042840 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:47:01.042980 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.043536 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.044355 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.044591 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.044671 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:01.044868 (+ 197us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:47:01.045750 (+ 882us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 05:47:01.045912 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 05:47:01.046049 17662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:01.042374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:35327 (local address 127.0.0.1:47928)
0926 05:47:01.042546 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:01.042557 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:01.042656 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:01.042838 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:01.042839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:01.042840 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:47:01.042905 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:01.042907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.043631 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.043633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:01.044261 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:01.044264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.044676 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.044677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:01.044739 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:01.044745 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:47:01.045857 (+ 1112us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 05:47:01.045955 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.247.62:35327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
W20260926 05:47:01.046079 17637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 05:47:01.046406 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17566
E20260926 05:47:01.057564 17663 webserver.cc:533] Webserver: sslize failed:
E20260926 05:47:01.057664 17663 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T05:47:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T05:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:01Z Disabled control of system clock
I20260926 05:47:01.092942 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33583
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33583
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/testchainca.cert with env {}
W20260926 05:47:01.176204 17667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:01.176476 17667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:01.176501 17667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:01.177795 17667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:01.177838 17667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:01.177850 17667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:01.177863 17667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:01.177878 17667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:47:01.177891 17667 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:47:01.177902 17667 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 05:47:01.179417 17667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:01.179625 17667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:01.179795 17667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:01.185025 17675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:01.185032 17673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:01.185099 17672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:01.185359 17667 server_base.cc:1034] running on GCE node
I20260926 05:47:01.185622 17667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:01.185936 17667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:01.187136 17667 hybrid_clock.cc:648] HybridClock initialized: now 1790401621187114 us; error 38 us; skew 500 ppm
I20260926 05:47:01.188540 17667 webserver.cc:460] Webserver started at http://127.11.247.62:44059/ using document root <none> and password file <none>
I20260926 05:47:01.188795 17667 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:01.188868 17667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:01.189060 17667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:01.190167 17667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "40497f7dd3cf4b6e89c58b11cd66c943"
format_stamp: "Formatted at 2026-09-26 05:47:01 on dist-test-slave-21p4"
server_key: "04704bbe22c0866d3ad7d6c71a2ecdef"
server_key_iv: "aee640df2b190979e021b0803342c4bb"
server_key_version: "encryptionkey@0"
I20260926 05:47:01.190553 17667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "40497f7dd3cf4b6e89c58b11cd66c943"
format_stamp: "Formatted at 2026-09-26 05:47:01 on dist-test-slave-21p4"
server_key: "04704bbe22c0866d3ad7d6c71a2ecdef"
server_key_iv: "aee640df2b190979e021b0803342c4bb"
server_key_version: "encryptionkey@0"
I20260926 05:47:01.192003 17667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:01.192776 17681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:01.192984 17667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:01.193102 17667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "40497f7dd3cf4b6e89c58b11cd66c943"
format_stamp: "Formatted at 2026-09-26 05:47:01 on dist-test-slave-21p4"
server_key: "04704bbe22c0866d3ad7d6c71a2ecdef"
server_key_iv: "aee640df2b190979e021b0803342c4bb"
server_key_version: "encryptionkey@0"
I20260926 05:47:01.193190 17667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:01.209411 17667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:01.209774 17667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:01.209944 17667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:01.214063 17667 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33583
I20260926 05:47:01.214133 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33583 every 8 connection(s)
I20260926 05:47:01.214491 17667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:01.215544 17734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:01.217871 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17667
I20260926 05:47:01.217983 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:01.218103 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943: Bootstrap starting.
I20260926 05:47:01.218165 12252 external_mini_cluster.cc:1428] Setting key 2e5a619408eaac4710fdfced3004e7c5
I20260926 05:47:01.218863 17734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:01.219187 17734 log.cc:824] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:01.220003 17734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943: No bootstrap required, opened a new log
I20260926 05:47:01.222100 17734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33583 } }
I20260926 05:47:01.222299 17734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:01.222349 17734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40497f7dd3cf4b6e89c58b11cd66c943, State: Initialized, Role: FOLLOWER
I20260926 05:47:01.222486 17734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33583 } }
I20260926 05:47:01.222579 17734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:01.222646 17734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:01.222699 17734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:01.223475 17734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33583 } }
I20260926 05:47:01.223598 17734 leader_election.cc:302] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40497f7dd3cf4b6e89c58b11cd66c943; no voters:
I20260926 05:47:01.223776 17734 leader_election.cc:288] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:01.223821 17739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:01.223990 17739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [term 1 LEADER]: Becoming Leader. State: Replica: 40497f7dd3cf4b6e89c58b11cd66c943, State: Running, Role: LEADER
I20260926 05:47:01.224066 17734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:01.224102 17739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33583 } }
I20260926 05:47:01.224596 17740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33583 } } }
I20260926 05:47:01.224591 17739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40497f7dd3cf4b6e89c58b11cd66c943. Latest consensus state: current_term: 1 leader_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40497f7dd3cf4b6e89c58b11cd66c943" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33583 } } }
I20260926 05:47:01.224782 17739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:01.224728 17740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:01.225288 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:01.218718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.11.247.62:33583)
0926 05:47:01.219017 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:01.219026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:01.219059 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:47:01.219564 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:01.219572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:01.219634 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:01.219758 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:01.219994 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.221013 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.222178 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.222843 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.222944 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:01.223671 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:01.223675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:01.223676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:01.224078 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:01.224123 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:01.224138 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:01.224736 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:01.224786 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 05:47:01.225024 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260926 05:47:01.225538 17748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:01.226145 17748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:01.226442 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:47:01.227995 17748 catalog_manager.cc:1347] Generated new cluster ID: db193ce195ff45e386d30202c09ed60f
I20260926 05:47:01.228049 17748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:01.254316 17748 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:01.254954 17748 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:01.261059 17748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40497f7dd3cf4b6e89c58b11cd66c943: Generated new TSK 0
I20260926 05:47:01.261288 17748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:47:01.357082 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:01.268668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.11.247.62:33583)
0926 05:47:01.268721 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:01.268723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:01.268805 (+ 82us) server_negotiation.cc:407] Connection header received
0926 05:47:01.268879 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:01.268880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:01.268895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:01.268983 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:47:01.269111 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.269919 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.270882 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.271211 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.271286 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:01.271461 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:47:01.356760 (+ 85299us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 05:47:01.356990 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57286: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 05:47:01.357170 17737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57286: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 05:47:01.357208 17763 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:01.268595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33583 (local address 127.0.0.1:57286)
0926 05:47:01.268724 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:01.268735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:01.268798 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:01.268987 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:01.268988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:01.268990 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:47:01.269043 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:01.269045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.270059 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.270060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:01.270795 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:01.270798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:01.271307 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:01.271308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:01.271351 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:01.271356 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:47:01.356947 (+ 85591us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 05:47:01.357081 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.247.62:33583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260926 05:47:01.357658 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17667
2026-09-26T05:47:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (292 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:01 dist-test-slave-21p4 krb5kdc[17775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:01 dist-test-slave-21p4 krb5kdc[17775](info): set up 2 sockets
Sep 26 05:47:01 dist-test-slave-21p4 krb5kdc[17775](info): commencing operation
krb5kdc: starting...
W20260926 05:47:02.615159 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:02 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:47:02.742147 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34537
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:02.829943 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:02.830245 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:02.830286 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:02.831703 17788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:02.831794 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:02.831818 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:02.832034 17788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:02.832113 17788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:02.834563 17788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:02.834964 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:02.835234 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:02.840670 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:02.840654 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:02.840654 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:02.841110 17788 server_base.cc:1034] running on GCE node
I20260926 05:47:02.841404 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:02.841791 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:02.842996 17788 hybrid_clock.cc:648] HybridClock initialized: now 1790401622842954 us; error 53 us; skew 500 ppm
Sep 26 05:47:02 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401622, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:02.845758 17788 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:47:02.846262 17788 webserver.cc:460] Webserver started at http://127.11.247.62:34823/ using document root <none> and password file <none>
I20260926 05:47:02.846519 17788 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:02.846611 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:02.846767 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:02.847985 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "ce835136bb3246d5b9dcf47fd335dd76"
format_stamp: "Formatted at 2026-09-26 05:47:02 on dist-test-slave-21p4"
server_key: "63a8c5d5606aa937738febc21bf4b484"
server_key_iv: "6461c13068e8933178db782c14f1fa65"
server_key_version: "encryptionkey@0"
I20260926 05:47:02.848379 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "ce835136bb3246d5b9dcf47fd335dd76"
format_stamp: "Formatted at 2026-09-26 05:47:02 on dist-test-slave-21p4"
server_key: "63a8c5d5606aa937738febc21bf4b484"
server_key_iv: "6461c13068e8933178db782c14f1fa65"
server_key_version: "encryptionkey@0"
I20260926 05:47:02.849815 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:47:02.850847 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:02.851125 17788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:02.851239 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "ce835136bb3246d5b9dcf47fd335dd76"
format_stamp: "Formatted at 2026-09-26 05:47:02 on dist-test-slave-21p4"
server_key: "63a8c5d5606aa937738febc21bf4b484"
server_key_iv: "6461c13068e8933178db782c14f1fa65"
server_key_version: "encryptionkey@0"
I20260926 05:47:02.851306 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:02.876021 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:02.878826 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:02.878984 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:02.884193 17788 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:34537
I20260926 05:47:02.884219 17855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:34537 every 8 connection(s)
I20260926 05:47:02.884902 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:02.885654 17856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:02.887987 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17788
I20260926 05:47:02.888082 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:02.888293 12252 external_mini_cluster.cc:1428] Setting key 4982efff4a40831d59a5c1e831de9eae
I20260926 05:47:02.888578 17856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76: Bootstrap starting.
I20260926 05:47:02.889658 17856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:02.890033 17856 log.cc:824] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:02.891093 17856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76: No bootstrap required, opened a new log
I20260926 05:47:02.893610 17856 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce835136bb3246d5b9dcf47fd335dd76" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34537 } }
I20260926 05:47:02.893821 17856 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:02.893857 17856 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce835136bb3246d5b9dcf47fd335dd76, State: Initialized, Role: FOLLOWER
I20260926 05:47:02.893990 17856 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce835136bb3246d5b9dcf47fd335dd76" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34537 } }
I20260926 05:47:02.894078 17856 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:02.894119 17856 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:02.894169 17856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:02.895022 17856 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce835136bb3246d5b9dcf47fd335dd76" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34537 } }
I20260926 05:47:02.895126 17856 leader_election.cc:302] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce835136bb3246d5b9dcf47fd335dd76; no voters:
I20260926 05:47:02.895385 17856 leader_election.cc:288] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:02.895411 17861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:02.895557 17861 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [term 1 LEADER]: Becoming Leader. State: Replica: ce835136bb3246d5b9dcf47fd335dd76, State: Running, Role: LEADER
I20260926 05:47:02.895649 17861 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce835136bb3246d5b9dcf47fd335dd76" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34537 } }
I20260926 05:47:02.895716 17856 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:02.895991 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce835136bb3246d5b9dcf47fd335dd76. Latest consensus state: current_term: 1 leader_uuid: "ce835136bb3246d5b9dcf47fd335dd76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce835136bb3246d5b9dcf47fd335dd76" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34537 } } }
I20260926 05:47:02.896073 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:02.896005 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce835136bb3246d5b9dcf47fd335dd76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce835136bb3246d5b9dcf47fd335dd76" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34537 } } }
I20260926 05:47:02.896114 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76 [sys.catalog]: This master's current role is: LEADER
Sep 26 05:47:02 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:02.897114 17865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:02.898094 17865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:02.900151 17865 catalog_manager.cc:1347] Generated new cluster ID: fe766c440f164c13b4f08dd590de9ad3
I20260926 05:47:02.900234 17865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:02.901734 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:02.888996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.11.247.62:34537)
0926 05:47:02.889181 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:02.889193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:02.889233 (+ 40us) server_negotiation.cc:407] Connection header received
0926 05:47:02.889923 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:02.889933 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:02.890010 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:02.890181 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:02.891049 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:02.892552 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:02.893405 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:02.894040 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:02.894161 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:02.897286 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:02.897292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:02.897293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:02.897310 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:02.899704 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:02.900227 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:02.900231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:02.900231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:02.900374 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:02.900636 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:02.900637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:02.900637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:02.901029 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:02.901184 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:02.901301 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 05:47:02.901475 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260926 05:47:02.904701 17865 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:02.905563 17865 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:02.913565 17865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce835136bb3246d5b9dcf47fd335dd76: Generated new TSK 0
I20260926 05:47:02.913822 17865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:47:02.942498 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34537
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:03.035737 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:03.036010 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:03.036054 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:03.037488 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:03.037563 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:03.037614 17882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:03.039332 17882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:03.039626 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:03.039871 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:03.045437 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:03.045495 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:03.045491 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:03.045651 17882 server_base.cc:1034] running on GCE node
I20260926 05:47:03.045969 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:03.046203 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:03.047435 17882 hybrid_clock.cc:648] HybridClock initialized: now 1790401623047421 us; error 43 us; skew 500 ppm
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.050483 17882 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:47:03.051056 17882 webserver.cc:460] Webserver started at http://127.11.247.1:34473/ using document root <none> and password file <none>
I20260926 05:47:03.051342 17882 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:03.051398 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:03.051594 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:03.052850 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "db6fe00e05dc413f9a8811887618b69b"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "842a9f362e4883e9361441937be5c8cb"
server_key_iv: "20c7d84dfb76daef0649cc55be097703"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.053269 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "db6fe00e05dc413f9a8811887618b69b"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "842a9f362e4883e9361441937be5c8cb"
server_key_iv: "20c7d84dfb76daef0649cc55be097703"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.054988 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 05:47:03.056025 17897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.056403 17882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:03.056491 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "db6fe00e05dc413f9a8811887618b69b"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "842a9f362e4883e9361441937be5c8cb"
server_key_iv: "20c7d84dfb76daef0649cc55be097703"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.056614 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:03.081248 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:03.084012 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:03.084157 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:03.084520 17882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:03.084887 17882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:03.084925 17882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.084949 17882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:03.084964 17882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.091568 17882 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:43605
I20260926 05:47:03.091616 18010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:43605 every 8 connection(s)
I20260926 05:47:03.092005 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:03.098893 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 17882
I20260926 05:47:03.099025 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:03.099261 12252 external_mini_cluster.cc:1428] Setting key ae00b51c0462a9c31c3e6bb951cfe2e1
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:03.105784 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.092944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:54863 (local address 127.11.247.62:34537)
0926 05:47:03.093017 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.093020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.093219 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:47:03.094217 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.094218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.094231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.094325 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.095970 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.096637 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.098448 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.098828 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.098939 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.102341 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:03.102348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:03.102348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:03.102357 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:03.104086 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.104631 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.104632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.104633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.104705 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.104990 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.104991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.104991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.105138 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:03.105221 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:03.105547 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 05:47:03.105673 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:47:03.106016 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.092820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34537 (local address 127.11.247.1:54863)
0926 05:47:03.093035 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:03.093092 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:03.094052 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:03.094512 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:03.094517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:03.095310 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:03.095562 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:03.095568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.096987 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.096989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:03.098279 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:03.098287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.099045 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.099048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:03.099265 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.100055 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:03.100068 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:03.102184 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:03.104212 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:03.104226 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:03.104230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:03.104544 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:03.104778 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:03.104779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:03.104779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:03.104916 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:03.105211 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:03.105218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:03.105394 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 05:47:03.105516 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 05:47:03.105740 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260926 05:47:03.106703 18011 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34537
I20260926 05:47:03.106834 18011 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:03.107050 18011 heartbeater.cc:511] Master 127.11.247.62:34537 requested a full tablet report, sending...
I20260926 05:47:03.107787 17820 ts_manager.cc:194] Registered new tserver with Master: db6fe00e05dc413f9a8811887618b69b (127.11.247.1:43605)
I20260926 05:47:03.108819 17820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:54863
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:47:03.130463 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34537
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:03.218113 18016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:03.218411 18016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:03.218452 18016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:03.219738 18016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:03.219822 18016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:03.219866 18016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:03.221482 18016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:03.221787 18016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:03.222043 18016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:47:03.227639 18016 server_base.cc:1034] running on GCE node
W20260926 05:47:03.227574 18024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:03.227556 18021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:03.227556 18022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:03.228075 18016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:03.228344 18016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:03.229521 18016 hybrid_clock.cc:648] HybridClock initialized: now 1790401623229490 us; error 48 us; skew 500 ppm
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.232281 18016 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:47:03.232827 18016 webserver.cc:460] Webserver started at http://127.11.247.2:43491/ using document root <none> and password file <none>
I20260926 05:47:03.233089 18016 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:03.233161 18016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:03.233336 18016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:03.234550 18016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "c4ba70ec0daa4454b7fa677773779988"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "506237b207356bff79b9dc6c740bb2fe"
server_key_iv: "efec05191d0d3f67bf6904c0e670b330"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.235028 18016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "c4ba70ec0daa4454b7fa677773779988"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "506237b207356bff79b9dc6c740bb2fe"
server_key_iv: "efec05191d0d3f67bf6904c0e670b330"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.236519 18016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:47:03.237501 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.237754 18016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:03.237869 18016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "c4ba70ec0daa4454b7fa677773779988"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "506237b207356bff79b9dc6c740bb2fe"
server_key_iv: "efec05191d0d3f67bf6904c0e670b330"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.237975 18016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:03.255275 18016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:03.257941 18016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:03.258170 18016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:03.258450 18016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:03.258878 18016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:03.258934 18016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.258982 18016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:03.259006 18016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.265709 18016 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:42117
I20260926 05:47:03.265751 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:42117 every 8 connection(s)
I20260926 05:47:03.266103 18016 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:47:03.266812 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18016
I20260926 05:47:03.266927 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:03.267127 12252 external_mini_cluster.cc:1428] Setting key 7a481d982d1f41d55393f6465e2198d4
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:47:03.277686 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.267043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:43425 (local address 127.11.247.62:34537)
0926 05:47:03.267112 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.267115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.267269 (+ 154us) server_negotiation.cc:407] Connection header received
0926 05:47:03.268422 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.268424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.268435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.268490 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.269257 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.269777 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.270922 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.271137 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.271229 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.274060 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:03.274062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:03.274063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:03.274073 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:03.275896 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.276454 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.276456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.276456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.276523 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.276839 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.276840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.276841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.276997 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:03.277085 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:03.277514 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 05:47:03.277613 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:47:03.277865 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.266933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34537 (local address 127.11.247.2:43425)
0926 05:47:03.267124 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:03.267182 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:03.268287 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:03.268580 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:03.268585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:03.268925 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:03.269135 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:03.269140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.269887 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.269889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:03.270811 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:03.270813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.271244 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.271245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:03.271395 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.272195 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:03.272209 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:03.273944 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:03.276018 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:03.276029 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:03.276032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:03.276335 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:03.276623 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:03.276624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:03.276624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:03.276759 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:03.277170 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:03.277179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:03.277355 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 05:47:03.277466 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:47:03.277616 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260926 05:47:03.278764 18145 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34537
I20260926 05:47:03.278901 18145 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:03.279192 18145 heartbeater.cc:511] Master 127.11.247.62:34537 requested a full tablet report, sending...
I20260926 05:47:03.280063 17820 ts_manager.cc:194] Registered new tserver with Master: c4ba70ec0daa4454b7fa677773779988 (127.11.247.2:42117)
I20260926 05:47:03.280575 17820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:43425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:47:03.299247 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34537
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:03.386102 18150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:03.386345 18150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:03.386373 18150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:03.387717 18150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:03.387770 18150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:03.387801 18150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:03.389447 18150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:03.389746 18150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:03.390017 18150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:03.395695 18156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:03.395864 18150 server_base.cc:1034] running on GCE node
W20260926 05:47:03.395707 18155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:03.395716 18158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:03.396315 18150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:03.396644 18150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:03.397828 18150 hybrid_clock.cc:648] HybridClock initialized: now 1790401623397810 us; error 50 us; skew 500 ppm
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.400801 18150 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:47:03.401357 18150 webserver.cc:460] Webserver started at http://127.11.247.3:37603/ using document root <none> and password file <none>
I20260926 05:47:03.401635 18150 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:03.401714 18150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:03.401885 18150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:03.403100 18150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "f9101b9e467644e097475e5eb63bb7d3"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "387e285342d7cbeb1809ee37779321d4"
server_key_iv: "2efebd0c1a912cc9afc1405dfa5db1e1"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.403501 18150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "f9101b9e467644e097475e5eb63bb7d3"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "387e285342d7cbeb1809ee37779321d4"
server_key_iv: "2efebd0c1a912cc9afc1405dfa5db1e1"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.404982 18150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:03.405925 18165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.406163 18150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:03.406296 18150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "f9101b9e467644e097475e5eb63bb7d3"
format_stamp: "Formatted at 2026-09-26 05:47:03 on dist-test-slave-21p4"
server_key: "387e285342d7cbeb1809ee37779321d4"
server_key_iv: "2efebd0c1a912cc9afc1405dfa5db1e1"
server_key_version: "encryptionkey@0"
I20260926 05:47:03.406381 18150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:03.416333 18150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:03.418958 18150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:03.419155 18150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:03.419442 18150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:03.419801 18150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:03.419867 18150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.419905 18150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:03.419942 18150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:03.426684 18150 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:35659
I20260926 05:47:03.426738 18278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:35659 every 8 connection(s)
I20260926 05:47:03.427136 18150 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:03.434913 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18150
I20260926 05:47:03.435017 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:03.435200 12252 external_mini_cluster.cc:1428] Setting key 1254027968fde1c13223c41d5db90bfe
I20260926 05:47:03.437489 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.427923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45683 (local address 127.11.247.62:34537)
0926 05:47:03.427997 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.428000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.428267 (+ 267us) server_negotiation.cc:407] Connection header received
0926 05:47:03.429247 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.429249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.429262 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.429331 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.430006 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.430574 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.431717 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.431972 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.432047 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.434389 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:03.434391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:03.434391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:03.434401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:03.436112 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.436528 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.436529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.436530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.436590 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.436792 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.436792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.436793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.436917 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:03.437008 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:03.437340 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 05:47:03.437416 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:47:03.437745 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.427908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34537 (local address 127.11.247.3:45683)
0926 05:47:03.428111 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:03.428150 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:03.429122 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:03.429400 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:03.429404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:03.429684 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:03.429889 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:03.429893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.430740 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.430741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:03.431610 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:03.431612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.432061 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.432062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:03.432174 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.432673 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:03.432682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:03.434272 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:03.436189 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:03.436196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:03.436198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:03.436439 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:03.436648 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:03.436649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:03.436649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:03.436738 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:03.437043 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:03.437049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:03.437194 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 05:47:03.437315 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 05:47:03.437482 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260926 05:47:03.438309 18279 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34537
I20260926 05:47:03.438530 18279 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:03.438783 18279 heartbeater.cc:511] Master 127.11.247.62:34537 requested a full tablet report, sending...
I20260926 05:47:03.439275 17820 ts_manager.cc:194] Registered new tserver with Master: f9101b9e467644e097475e5eb63bb7d3 (127.11.247.3:35659)
I20260926 05:47:03.439695 17820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:45683
I20260926 05:47:03.447155 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/test-admin.keytab.
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.464141 12252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 05:47:03.466041 18286 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:03.465109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.11.247.1:43605)
0926 05:47:03.465288 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.465293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.465306 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:47:03.465335 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.465337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.465350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.465431 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.465952 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60378: BlockingRecv error: recv got EOF from 127.0.0.1:60378 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
W20260926 05:47:03.466082 12252 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.467216 12252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.468101 12252 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:47:03.476722 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.468696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.11.247.1:43605)
0926 05:47:03.468759 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.468761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.468771 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:03.468816 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.468817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.468826 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.468894 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.469281 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.470051 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.471016 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.471367 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.471441 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.473774 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:03.473785 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:03.473785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:03.473793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:03.475535 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.475958 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.475961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.475961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.476021 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.476197 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.476198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.476198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.476338 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:03.476479 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:03.476568 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 05:47:03.476640 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 05:47:03.479024 18286 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:03.478227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.11.247.1:43605)
0926 05:47:03.478293 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.478295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.478379 (+ 84us) server_negotiation.cc:407] Connection header received
0926 05:47:03.478463 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.478464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.478474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.478534 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.478964 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60394: BlockingRecv error: recv got EOF from 127.0.0.1:60394 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 05:47:03.479171 12252 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.480170 12252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.481035 12252 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:47:03.488535 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.481584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.11.247.1:43605)
0926 05:47:03.481641 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.481643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.481715 (+ 72us) server_negotiation.cc:407] Connection header received
0926 05:47:03.481793 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.481794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.481811 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.481871 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.482295 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.482863 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.483571 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.483771 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.483847 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.485903 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:03.485905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:03.485905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:03.485912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:03.487366 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.487743 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.487744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.487745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.487804 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.488015 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.488016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.488016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.488125 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:03.488230 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:03.488335 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 05:47:03.488415 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 05:47:03.490300 18286 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:03.489703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.11.247.1:43605)
0926 05:47:03.489738 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.489740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.489779 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:47:03.489878 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.489878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.489886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.489935 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.490246 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60422: BlockingRecv error: recv got EOF from 127.0.0.1:60422 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 05:47:03.490407 12252 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401595249435-12252-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.491591 12252 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:03.492589 12252 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.1@KRBTEST.COM
I20260926 05:47:03.500535 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:03.493189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.11.247.1:43605)
0926 05:47:03.493255 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:03.493258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:03.493266 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:03.493353 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:03.493354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:03.493364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:03.493420 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:03.493802 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.494303 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.495108 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:03.495325 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:03.495402 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:03.497829 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:03.497831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:03.497831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:03.497839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:03.499379 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.499751 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.499752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.499753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.499812 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:03.500022 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:03.500023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:03.500023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:03.500135 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:03.500239 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:03.500347 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 05:47:03.500423 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:47:03.501485 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17882
I20260926 05:47:03.507501 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18016
I20260926 05:47:03.512190 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18150
I20260926 05:47:03.516745 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 17788
2026-09-26T05:47:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1973 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[18299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[18299](info): set up 2 sockets
Sep 26 05:47:03 dist-test-slave-21p4 krb5kdc[18299](info): commencing operation
krb5kdc: starting...
W20260926 05:47:04.604336 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:04 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab.
Entry for principal oryx/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab.
I20260926 05:47:04.747610 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42167
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab
--principal=oryx/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:04.838505 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:04.838794 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:04.838821 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:04.840134 18312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:04.840181 18312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:04.840195 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:04.840209 18312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:04.840220 18312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:04.841854 18312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab
--principal=oryx/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:04.842105 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:04.842305 18312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:04.848155 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:04.848181 18318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:04.848150 18317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:04.848989 18312 server_base.cc:1034] running on GCE node
I20260926 05:47:04.849288 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:04.849635 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:04.851008 18312 hybrid_clock.cc:648] HybridClock initialized: now 1790401624850788 us; error 67 us; skew 500 ppm
Sep 26 05:47:04 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:04.854173 18312 init.cc:377] Logged in from keytab as oryx/127.11.247.62@KRBTEST.COM (short username oryx)
I20260926 05:47:04.854871 18312 webserver.cc:460] Webserver started at http://127.11.247.62:40287/ using document root <none> and password file <none>
I20260926 05:47:04.855170 18312 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:04.855247 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:04.855397 18312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:04.856567 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "a9cf3850e4ed4be1a1462dbb86082e5b"
format_stamp: "Formatted at 2026-09-26 05:47:04 on dist-test-slave-21p4"
server_key: "e1fffd1894bb530c12b263351c918662"
server_key_iv: "e6b8ad4e763b2b290500ae8767d0ea29"
server_key_version: "encryptionkey@0"
I20260926 05:47:04.856973 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "a9cf3850e4ed4be1a1462dbb86082e5b"
format_stamp: "Formatted at 2026-09-26 05:47:04 on dist-test-slave-21p4"
server_key: "e1fffd1894bb530c12b263351c918662"
server_key_iv: "e6b8ad4e763b2b290500ae8767d0ea29"
server_key_version: "encryptionkey@0"
I20260926 05:47:04.858525 18312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:47:04.859551 18327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:04.859912 18312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:04.860047 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "a9cf3850e4ed4be1a1462dbb86082e5b"
format_stamp: "Formatted at 2026-09-26 05:47:04 on dist-test-slave-21p4"
server_key: "e1fffd1894bb530c12b263351c918662"
server_key_iv: "e6b8ad4e763b2b290500ae8767d0ea29"
server_key_version: "encryptionkey@0"
I20260926 05:47:04.860139 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:04.871344 18312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:04.873996 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:04.874233 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:04.879168 18312 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:42167
I20260926 05:47:04.879223 18379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:42167 every 8 connection(s)
I20260926 05:47:04.879630 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:04.880740 18380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:04.883701 18380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Bootstrap starting.
I20260926 05:47:04.884143 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18312
I20260926 05:47:04.884255 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:04.884523 12252 external_mini_cluster.cc:1428] Setting key cbd5d732be9179263898491f36bbac48
I20260926 05:47:04.884763 18380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:04.885118 18380 log.cc:824] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:04.885931 18380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: No bootstrap required, opened a new log
I20260926 05:47:04.888443 18380 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } }
I20260926 05:47:04.888649 18380 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:04.888686 18380 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9cf3850e4ed4be1a1462dbb86082e5b, State: Initialized, Role: FOLLOWER
I20260926 05:47:04.888816 18380 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } }
I20260926 05:47:04.888919 18380 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:04.888950 18380 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:04.888996 18380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:04.889679 18380 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } }
I20260926 05:47:04.889827 18380 leader_election.cc:302] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9cf3850e4ed4be1a1462dbb86082e5b; no voters:
I20260926 05:47:04.890036 18380 leader_election.cc:288] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:04.890183 18385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:04.890347 18380 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:04.890360 18385 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 LEADER]: Becoming Leader. State: Replica: a9cf3850e4ed4be1a1462dbb86082e5b, State: Running, Role: LEADER
I20260926 05:47:04.890511 18385 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } }
I20260926 05:47:04.890868 18385 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9cf3850e4ed4be1a1462dbb86082e5b. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } }
I20260926 05:47:04.890969 18385 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:04.890873 18387 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } }
I20260926 05:47:04.891111 18387 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:04.891316 18392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:04.892130 18392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:47:04 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:04.895053 18392 catalog_manager.cc:1347] Generated new cluster ID: f491556a2f234ef88e8eb1db84ac071e
I20260926 05:47:04.895120 18392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:04.897312 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:04.885134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.11.247.62:42167)
0926 05:47:04.885378 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:04.885389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:04.885429 (+ 40us) server_negotiation.cc:407] Connection header received
0926 05:47:04.885997 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:04.886005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:04.886099 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:04.886241 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:04.887144 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:04.888092 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:04.889202 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:04.889679 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:04.889813 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:04.893143 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:04.893148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:04.893149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:04.893172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:04.895466 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:04.895915 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:04.895918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:04.895919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:04.896045 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:04.896295 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:04.896296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:04.896297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:04.896581 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:04.896781 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:04.896868 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:47:04.897072 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20260926 05:47:04.909783 18392 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:04.910799 18392 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:04.918382 18392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Generated new TSK 0
I20260926 05:47:04.918671 18392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab.
Entry for principal oryx/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab.
I20260926 05:47:04.955183 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42167
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:05.047967 18406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:05.048243 18406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:05.048290 18406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:05.049736 18406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:05.049937 18406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:05.049996 18406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:05.052069 18406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:05.052408 18406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:05.052670 18406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:05.058841 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.058776 18412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.058904 18406 server_base.cc:1034] running on GCE node
W20260926 05:47:05.058770 18411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.059321 18406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:05.059612 18406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:05.060765 18406 hybrid_clock.cc:648] HybridClock initialized: now 1790401625060744 us; error 35 us; skew 500 ppm
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:05.064579 18406 init.cc:377] Logged in from keytab as oryx/127.11.247.1@KRBTEST.COM (short username oryx)
I20260926 05:47:05.065212 18406 webserver.cc:460] Webserver started at http://127.11.247.1:34525/ using document root <none> and password file <none>
I20260926 05:47:05.065485 18406 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.065562 18406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.065709 18406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:05.067167 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "8e46366f58cb4b23afecfd2e39986d2b"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "5a20bff365ee5454deb29950840ea754"
server_key_iv: "d43a42731df9c06adb7b8e8c788cb3ca"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.067593 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "8e46366f58cb4b23afecfd2e39986d2b"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "5a20bff365ee5454deb29950840ea754"
server_key_iv: "d43a42731df9c06adb7b8e8c788cb3ca"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.069164 18406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:05.070104 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.070389 18406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:05.070528 18406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "8e46366f58cb4b23afecfd2e39986d2b"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "5a20bff365ee5454deb29950840ea754"
server_key_iv: "d43a42731df9c06adb7b8e8c788cb3ca"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.070689 18406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.091593 18406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:05.094165 18406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:05.094349 18406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:05.094658 18406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:05.095024 18406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:05.095088 18406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.095130 18406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:05.095184 18406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.101466 18406 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:33433
I20260926 05:47:05.101508 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:33433 every 8 connection(s)
I20260926 05:47:05.101930 18406 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:05.111997 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18406
I20260926 05:47:05.112164 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:05.112396 12252 external_mini_cluster.cc:1428] Setting key 700a95d94fc47e7ef498b37aae248d7e
I20260926 05:47:05.113054 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.102784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:51793 (local address 127.11.247.62:42167)
0926 05:47:05.102863 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.102865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.103119 (+ 254us) server_negotiation.cc:407] Connection header received
0926 05:47:05.103660 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.103661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.103673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.103732 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.104808 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.105376 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.106736 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.106974 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.107042 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.109465 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.109467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.109467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.109475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.111129 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.111638 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.111639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.111639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.111714 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.112202 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.112204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.112204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.112391 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.112496 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.112829 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.112959 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:47:05.113281 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.102781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:51793)
0926 05:47:05.103002 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.103043 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.103553 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.103816 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.103820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.104482 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.104697 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.104702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.105485 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.105486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.106518 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.106521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.107085 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.107086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.107221 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.107844 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.107853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.109346 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.111230 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.111241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.111244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.111534 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.111866 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.111866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.111867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.111996 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.112490 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.112499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.112663 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.112805 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.113006 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":119,"threads_started":1}
I20260926 05:47:05.113865 18535 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42167
I20260926 05:47:05.113970 18535 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:05.114137 18535 heartbeater.cc:511] Master 127.11.247.62:42167 requested a full tablet report, sending...
I20260926 05:47:05.114775 18344 ts_manager.cc:194] Registered new tserver with Master: 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433)
I20260926 05:47:05.115422 18344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:51793
WARNING: no policy specified for oryx/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab.
Entry for principal oryx/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab.
I20260926 05:47:05.142962 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42167
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:05.229384 18540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:05.229648 18540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:05.229692 18540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:05.231073 18540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:05.231153 18540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:05.231205 18540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:05.232820 18540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:05.233137 18540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:05.233414 18540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:47:05.239620 18540 server_base.cc:1034] running on GCE node
W20260926 05:47:05.239563 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.239609 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.239578 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.240104 18540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:05.240387 18540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:05.241750 18540 hybrid_clock.cc:648] HybridClock initialized: now 1790401625241726 us; error 36 us; skew 500 ppm
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:05.244763 18540 init.cc:377] Logged in from keytab as oryx/127.11.247.2@KRBTEST.COM (short username oryx)
I20260926 05:47:05.245316 18540 webserver.cc:460] Webserver started at http://127.11.247.2:37201/ using document root <none> and password file <none>
I20260926 05:47:05.245572 18540 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.245647 18540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.245803 18540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:05.247044 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "18e03719a5bb43f8ade3cfb83377062d"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "6714bfad35531f6a2086dfe22564802f"
server_key_iv: "7e7d9885b3bb729a3953723a700606bc"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.247431 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "18e03719a5bb43f8ade3cfb83377062d"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "6714bfad35531f6a2086dfe22564802f"
server_key_iv: "7e7d9885b3bb729a3953723a700606bc"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.248888 18540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:05.249763 18555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.250037 18540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:05.250171 18540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "18e03719a5bb43f8ade3cfb83377062d"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "6714bfad35531f6a2086dfe22564802f"
server_key_iv: "7e7d9885b3bb729a3953723a700606bc"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.250262 18540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.271207 18540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:05.273823 18540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:05.273998 18540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:05.274271 18540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:05.274674 18540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:05.274729 18540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.274791 18540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:05.274816 18540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.280710 18540 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:38785
I20260926 05:47:05.280794 18668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:38785 every 8 connection(s)
I20260926 05:47:05.281175 18540 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:05.289203 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18540
I20260926 05:47:05.289325 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:05.289548 12252 external_mini_cluster.cc:1428] Setting key 4d3e95871f7935400aacf5c80f4eaa05
I20260926 05:47:05.293349 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.282016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:54353 (local address 127.11.247.62:42167)
0926 05:47:05.282107 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.282109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.282273 (+ 164us) server_negotiation.cc:407] Connection header received
0926 05:47:05.282969 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.282971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.282982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.283054 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.284146 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.284705 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.286073 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.286278 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.286350 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.289225 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.289226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.289227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.289276 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.291492 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.292075 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.292076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.292077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.292154 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.292535 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.292537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.292537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.292673 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.292740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.293189 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.293275 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 05:47:05.293615 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.281961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:54353)
0926 05:47:05.282137 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.282180 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.282796 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.283167 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.283171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.283810 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.284032 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.284036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.284875 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.284876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.285808 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.285812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.286388 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.286390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.286524 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.287367 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.287377 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.288984 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.291623 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.291636 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.291640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.291987 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.292243 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.292244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.292245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.292435 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.292752 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.292758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.292971 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.293119 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.293329 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260926 05:47:05.294093 18669 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42167
I20260926 05:47:05.294224 18669 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:05.294431 18669 heartbeater.cc:511] Master 127.11.247.62:42167 requested a full tablet report, sending...
I20260926 05:47:05.294984 18344 ts_manager.cc:194] Registered new tserver with Master: 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:05.295562 18344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:54353
WARNING: no policy specified for oryx/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab.
Entry for principal oryx/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab.
I20260926 05:47:05.322019 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42167
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:05.416190 18674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:05.416477 18674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:05.416512 18674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:05.418308 18674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:05.418367 18674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:05.418401 18674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:05.420356 18674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:05.420605 18674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:05.420812 18674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:05.426407 18682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.426407 18679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.426450 18680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.426784 18674 server_base.cc:1034] running on GCE node
I20260926 05:47:05.427001 18674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:05.427285 18674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:05.428483 18674 hybrid_clock.cc:648] HybridClock initialized: now 1790401625428455 us; error 46 us; skew 500 ppm
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:05.431569 18674 init.cc:377] Logged in from keytab as oryx/127.11.247.3@KRBTEST.COM (short username oryx)
I20260926 05:47:05.432160 18674 webserver.cc:460] Webserver started at http://127.11.247.3:36979/ using document root <none> and password file <none>
I20260926 05:47:05.432444 18674 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.432523 18674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.432703 18674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:05.433892 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "1589b65ad8544232b5c76d9cb9d44c64"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "dfdb04af82a3dfa22efa2a4c47aaf7bf"
server_key_iv: "1436a46ccb876abde4a1b0810137c9a7"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.434311 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "1589b65ad8544232b5c76d9cb9d44c64"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "dfdb04af82a3dfa22efa2a4c47aaf7bf"
server_key_iv: "1436a46ccb876abde4a1b0810137c9a7"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.436007 18674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 05:47:05.436906 18689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.437093 18674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:05.437207 18674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "1589b65ad8544232b5c76d9cb9d44c64"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "dfdb04af82a3dfa22efa2a4c47aaf7bf"
server_key_iv: "1436a46ccb876abde4a1b0810137c9a7"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.437292 18674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.449064 18674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:05.451695 18674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:05.451892 18674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:05.452159 18674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:05.452497 18674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:05.452551 18674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.452595 18674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:05.452625 18674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.458837 18674 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:40379
I20260926 05:47:05.458925 18802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:40379 every 8 connection(s)
I20260926 05:47:05.459277 18674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:05.468317 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18674
I20260926 05:47:05.468439 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:05.468679 12252 external_mini_cluster.cc:1428] Setting key f5f12e85a889f58804d000666d80dd95
I20260926 05:47:05.469364 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.460096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:42871 (local address 127.11.247.62:42167)
0926 05:47:05.460148 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.460150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.460383 (+ 233us) server_negotiation.cc:407] Connection header received
0926 05:47:05.460870 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.460871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.460883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.460953 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.461896 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.462412 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.463568 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.463769 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.463839 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.466224 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.466226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.466226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.466235 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.467761 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.468170 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.468172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.468172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.468240 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.468565 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.468566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.468567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.468728 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.468802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.469177 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.469265 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:47:05.469589 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.460093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:42871)
0926 05:47:05.460252 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.460290 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.460778 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.461008 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.461012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.461582 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.461789 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.461794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.462536 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.462538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.463484 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.463487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.463912 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.463913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.464035 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.464637 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.464646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.466116 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.467842 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.467851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.467854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.468091 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.468314 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.468314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.468315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.468478 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.468835 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.468843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.469028 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.469154 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.469329 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260926 05:47:05.470108 18803 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42167
I20260926 05:47:05.470213 18803 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:05.470420 18803 heartbeater.cc:511] Master 127.11.247.62:42167 requested a full tablet report, sending...
I20260926 05:47:05.470929 18344 ts_manager.cc:194] Registered new tserver with Master: 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3:40379)
I20260926 05:47:05.471447 18344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:42871
I20260926 05:47:05.480965 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.247.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.61.keytab.
Entry for principal oryx/127.11.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.61.keytab.
I20260926 05:47:05.510566 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:46189
--webserver_interface=127.11.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.61.keytab
--principal=oryx/127.11.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42167,127.11.247.61:46189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:05.596186 18808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:05.596469 18808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:05.596510 18808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:05.597747 18808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:05.597833 18808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:05.597858 18808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:05.597882 18808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:05.597910 18808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:05.599561 18808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42167,127.11.247.61:46189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.61.keytab
--principal=oryx/127.11.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:46189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
W20260926 05:47:05.599763 18808 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.247.62:42167,127.11.247.61:46189'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 05:47:05.599946 18808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:05.600200 18808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:05.606738 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.606730 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.606918 18808 server_base.cc:1034] running on GCE node
W20260926 05:47:05.606722 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.607302 18808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:05.607653 18808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:05.608852 18808 hybrid_clock.cc:648] HybridClock initialized: now 1790401625608810 us; error 65 us; skew 500 ppm
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:05.612385 18808 init.cc:377] Logged in from keytab as oryx/127.11.247.61@KRBTEST.COM (short username oryx)
I20260926 05:47:05.613018 18808 webserver.cc:460] Webserver started at http://127.11.247.61:46633/ using document root <none> and password file <none>
I20260926 05:47:05.613308 18808 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.613386 18808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.613560 18808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:05.614867 18808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/instance:
uuid: "93e906df8f6849c293efc589a8d96490"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "3c4f2dabafb5c91e092e4047244512a7"
server_key_iv: "ce265fd8302a1f3c759f350c7c522160"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.615285 18808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal/instance:
uuid: "93e906df8f6849c293efc589a8d96490"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "3c4f2dabafb5c91e092e4047244512a7"
server_key_iv: "ce265fd8302a1f3c759f350c7c522160"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.616971 18808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:05.617944 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.618180 18808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:05.618306 18808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
uuid: "93e906df8f6849c293efc589a8d96490"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "3c4f2dabafb5c91e092e4047244512a7"
server_key_iv: "ce265fd8302a1f3c759f350c7c522160"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.618397 18808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.642463 18808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:05.644999 18808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:05.645175 18808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:05.649370 18808 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.61:46189
I20260926 05:47:05.649413 18875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.61:46189 every 8 connection(s)
I20260926 05:47:05.649780 18808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
I20260926 05:47:05.651176 18876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:05.652923 18876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:47:05.656509 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18808
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.61@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
I20260926 05:47:05.664278 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.653727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.11.247.62:42167)
0926 05:47:05.653850 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.653853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.654096 (+ 243us) server_negotiation.cc:407] Connection header received
0926 05:47:05.654756 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.654757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.654774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.654827 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.655571 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.656125 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.657611 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.657806 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.657861 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.660436 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.660438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.660438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.660447 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.662210 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.662845 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.662847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.662847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.662915 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.663241 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.663242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.663242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.663476 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.663582 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.664053 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.664158 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 05:47:05.664484 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.653716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54354)
0926 05:47:05.653967 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.654009 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.654562 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.654947 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.654951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.655242 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.655464 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.655469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.656245 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.656246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.657526 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.657529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.657899 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.657900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.658116 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.658834 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.658844 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.660338 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.662349 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.662361 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.662363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.662747 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.662990 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.662991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.662991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.663175 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.663575 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.663582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.663771 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.663942 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.664201 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":121,"threads_started":1}
I20260926 05:47:05.665171 18876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:47:05.667078 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.657117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.11.247.61:46189)
0926 05:47:05.657261 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.657265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.657335 (+ 70us) server_negotiation.cc:407] Connection header received
0926 05:47:05.657392 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.657394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.657414 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.657478 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.658310 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.659223 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.660074 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.660503 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.660581 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.663424 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.663431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.663431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.663438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.665667 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.666090 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.666093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.666094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.666165 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.666390 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.666391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.666392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.666549 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.666794 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.666851 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.666953 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.61@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
I20260926 05:47:05.672917 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.665499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37006)
0926 05:47:05.665575 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.665586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.665635 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.665892 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.665893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.666037 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.666084 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.666086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.666833 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.666834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.667472 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.667474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.667933 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.667933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.667999 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.668357 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.668361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.669501 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.671857 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.671859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.671859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.672064 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.672282 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.672283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.672284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.672344 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.672619 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.672620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.672663 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.672737 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.672840 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":3}
I20260926 05:47:05.673058 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.665591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.11.247.61:46189)
0926 05:47:05.665769 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.665771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.665777 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:47:05.665795 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.665796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.665813 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.665892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.666170 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.666749 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.667560 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.667850 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.667934 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.669599 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.669601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.669601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.669607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.671669 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.672136 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.672137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.672137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.672192 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.672402 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.672403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.672403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.672506 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.672574 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.672755 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.672944 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260926 05:47:05.674465 18876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Bootstrap starting.
I20260926 05:47:05.675202 18876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:05.675500 18876 log.cc:824] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:05.676374 18876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: No bootstrap required, opened a new log
I20260926 05:47:05.677855 18876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } }
I20260926 05:47:05.677995 18876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:05.678020 18876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e906df8f6849c293efc589a8d96490, State: Initialized, Role: FOLLOWER
I20260926 05:47:05.678181 18876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } }
I20260926 05:47:05.678516 18876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:05.678475 18883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } } }
I20260926 05:47:05.678658 18883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:47:05.680392 18894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:47:05.680461 18894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:47:05.681988 18808 master_runner.cc:428] Detected that this master 93e906df8f6849c293efc589a8d96490 is joining an existing cluster
I20260926 05:47:05.682039 18808 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.247.61:46189
I20260926 05:47:05.690474 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.683224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.11.247.62:42167)
0926 05:47:05.683275 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.683278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.683440 (+ 162us) server_negotiation.cc:407] Connection header received
0926 05:47:05.683571 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.683573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.683588 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.683657 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.684171 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.684916 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.685636 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.685858 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.685954 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.687295 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.687297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.687297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.687306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.689165 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689589 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.689590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.689591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.689647 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689868 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.689869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.689869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.690019 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.690066 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.690264 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.690358 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:47:05.690726 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.683217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54370)
0926 05:47:05.683367 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.683377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.683470 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.683723 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.683724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.683997 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.684079 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.684081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.684972 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.684973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.685551 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.685553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.686028 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.686029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.686087 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.686715 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.686720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.687158 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.689265 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.689266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.689514 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.689723 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.689724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.689787 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.690110 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.690113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.690157 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.690227 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.690348 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260926 05:47:05.691226 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.683241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.11.247.61:46189)
0926 05:47:05.683290 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.683292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.683476 (+ 184us) server_negotiation.cc:407] Connection header received
0926 05:47:05.683565 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.683566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.683575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.683619 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.684021 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.684816 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.685918 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.686194 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.686262 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.687691 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.687692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.687692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.687698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.689312 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689798 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.689798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.689799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.689846 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.690060 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.690062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.690063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.690212 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.690292 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.690526 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.691130 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":23}
I20260926 05:47:05.691644 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.683300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37008)
0926 05:47:05.683423 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.683432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.683495 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.683622 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.683623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.683876 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.683943 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.683944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.684913 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.684914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.685824 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.685827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.686671 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.686672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.686754 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.687231 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.687237 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.687611 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.689402 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.689403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.689691 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.689922 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.689923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.689923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.689988 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.690316 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.690317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.690393 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.691403 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.691544 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.62@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
I20260926 05:47:05.700985 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.692996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37020)
0926 05:47:05.693201 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.693251 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.693337 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.693482 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.693485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.693859 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.693934 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.693937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.694632 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.694633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.695587 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.695589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.695946 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.695947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.696037 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.696536 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.696542 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.698114 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.699808 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.699816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.699818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.700021 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.700232 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.700233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.700234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.700304 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.700570 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.700578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.700649 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.700734 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.700872 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
I20260926 05:47:05.700976 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.693084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.11.247.61:46189)
0926 05:47:05.693124 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.693126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.693309 (+ 183us) server_negotiation.cc:407] Connection header received
0926 05:47:05.693419 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.693420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.693429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.693472 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.694034 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.694508 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.695678 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.695859 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.695925 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.698211 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.698212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.698213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.698220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.699658 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.700102 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.700103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.700103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.700165 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.700370 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.700371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.700371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.700489 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.700564 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.700786 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.700868 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:47:05.701769 18344 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.247.61:46189: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "93e906df8f6849c293efc589a8d96490"
member_type: NON_VOTER
last_known_addr {
host: "127.11.247.61"
port: 46189
}
attrs {
promote: true
}
}
dest_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b"
cas_config_opid_index: -1
I20260926 05:47:05.702117 18344 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } }
I20260926 05:47:05.702638 18904 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 93e906df8f6849c293efc589a8d96490 (127.11.247.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } }
I20260926 05:47:05.703058 18850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:05.703684 18904 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.247.61:46189
I20260926 05:47:05.703893 18903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } }
I20260926 05:47:05.703986 18808 master.cc:537] Master@127.11.247.61:46189 shutting down...
I20260926 05:47:05.704000 18850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Refusing update from remote peer a9cf3850e4ed4be1a1462dbb86082e5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 05:47:05.704072 18903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:05.704449 18903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:47:05.705466 18331 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 05:47:05.706048 18331 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b -> Peer 93e906df8f6849c293efc589a8d96490 (127.11.247.61:46189): Couldn't send request to peer 93e906df8f6849c293efc589a8d96490. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:47:05.707326 18808 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:05.707432 18808 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:05.707484 18808 tablet_replica.cc:330] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: stopping tablet replica
I20260926 05:47:05.710208 18808 master.cc:552] Master@127.11.247.61:46189 shutdown complete.
I20260926 05:47:05.710281 18808 master_runner.cc:305] Clearing existing system tablet
I20260926 05:47:05.710418 18808 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.710494 18808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.711366 18808 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:47:05.712674 18908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.712885 18808 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.712954 18808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:47:05.712983 18808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
uuid: "93e906df8f6849c293efc589a8d96490"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "3c4f2dabafb5c91e092e4047244512a7"
server_key_iv: "ce265fd8302a1f3c759f350c7c522160"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.714010 18808 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:05.715704 18808 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 05:47:05.715806 18808 log.cc:1196] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 05:47:05.716029 18808 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Deleting consensus metadata
I20260926 05:47:05.716166 18808 master_runner.cc:315] Copying system tablet from 127.11.247.62:42167
I20260926 05:47:05.716897 18808 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: tablet copy: Beginning tablet copy session from remote peer at address 127.11.247.62:42167
I20260926 05:47:05.724323 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.717257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54376)
0926 05:47:05.717440 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.717449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.717535 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.717687 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.717687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.717964 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.718035 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.718038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.719044 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.719046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.719805 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.719808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.720366 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.720367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.720439 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.720867 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.720872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.721317 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.723090 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.723092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.723092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.723338 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.723571 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.723571 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.723572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.723641 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.723948 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.723949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.724003 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.724086 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.724206 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260926 05:47:05.724356 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.717330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.11.247.62:42167)
0926 05:47:05.717389 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.717391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.717519 (+ 128us) server_negotiation.cc:407] Connection header received
0926 05:47:05.717600 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.717601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.717612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.717683 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.718122 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.718952 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.719905 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.720208 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.720292 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.721404 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.721405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.721406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.721412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.723002 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.723426 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.723427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.723428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.723490 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.723729 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.723730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.723731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.723879 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.723940 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.724122 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.724206 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:47:05.724633 18364 tablet_copy_service.cc:140] P a9cf3850e4ed4be1a1462dbb86082e5b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 93e906df8f6849c293efc589a8d96490 ({username='oryx', principal='oryx/127.11.247.61@KRBTEST.COM'} at 127.0.0.1:54376)
I20260926 05:47:05.724720 18364 tablet_copy_service.cc:161] P a9cf3850e4ed4be1a1462dbb86082e5b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 93e906df8f6849c293efc589a8d96490 at {username='oryx', principal='oryx/127.11.247.61@KRBTEST.COM'} at 127.0.0.1:54376: session id = 93e906df8f6849c293efc589a8d96490-00000000000000000000000000000000
I20260926 05:47:05.725806 18364 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:47:05.726496 18808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 05:47:05.728031 18808 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: tablet copy: Starting download of 0 data blocks...
I20260926 05:47:05.728207 18808 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: tablet copy: Starting download of 1 WAL segments...
I20260926 05:47:05.729185 18808 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:47:05.730257 18364 tablet_copy_service.cc:342] P a9cf3850e4ed4be1a1462dbb86082e5b: Request end of tablet copy session 93e906df8f6849c293efc589a8d96490-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.247.61@KRBTEST.COM'} at 127.0.0.1:54376
I20260926 05:47:05.730329 18364 tablet_copy_service.cc:433] P a9cf3850e4ed4be1a1462dbb86082e5b: ending tablet copy session 93e906df8f6849c293efc589a8d96490-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 93e906df8f6849c293efc589a8d96490
I20260926 05:47:05.733417 18808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:05.734862 18918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.734901 18915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.735118 18916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.735188 18808 server_base.cc:1034] running on GCE node
I20260926 05:47:05.735327 18808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:05.735469 18808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:05.736601 18808 hybrid_clock.cc:648] HybridClock initialized: now 1790401625736581 us; error 39 us; skew 500 ppm
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:05.738005 18808 init.cc:377] Logged in from keytab as oryx/127.11.247.61@KRBTEST.COM (short username oryx)
I20260926 05:47:05.738297 18808 webserver.cc:460] Webserver started at http://127.11.247.61:46083/ using document root <none> and password file <none>
I20260926 05:47:05.738451 18808 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.738482 18808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.739229 18808 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.739864 18925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.739989 18808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:47:05.740037 18808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
uuid: "93e906df8f6849c293efc589a8d96490"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "3c4f2dabafb5c91e092e4047244512a7"
server_key_iv: "ce265fd8302a1f3c759f350c7c522160"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.740098 18808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.749397 18808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:05.751062 18808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:05.754566 18808 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.61:46189
I20260926 05:47:05.754841 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.61:46189 every 8 connection(s)
I20260926 05:47:05.755064 18808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
I20260926 05:47:05.755844 18978 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:47:05.756294 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Bootstrap starting.
I20260926 05:47:05.758803 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:47:05.759032 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Bootstrap complete.
I20260926 05:47:05.759238 18978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } }
I20260926 05:47:05.759313 18978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93e906df8f6849c293efc589a8d96490, State: Initialized, Role: LEARNER
I20260926 05:47:05.759341 18978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } }
I20260926 05:47:05.759564 18978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:05.759545 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } }
I20260926 05:47:05.759632 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:47:05.761023 18992 catalog_manager.cc:1259] Loaded cluster ID: f491556a2f234ef88e8eb1db84ac071e
I20260926 05:47:05.761073 18992 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: loading cluster ID for follower catalog manager: success
I20260926 05:47:05.762109 18992 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: acquiring CA information for follower catalog manager: success
I20260926 05:47:05.762475 18992 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:47:05.775655 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.768668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.11.247.61:46189)
0926 05:47:05.768811 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.768813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.768820 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:05.768846 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.768846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.768859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.768953 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.769522 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.770170 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.770995 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.771198 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.771286 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.772714 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.772716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.772717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.772725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.774329 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.774832 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.774833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.774834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.774897 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.775099 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.775100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.775100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.775221 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.775352 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.775442 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.775536 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for oryx/127.11.247.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.60.keytab.
Entry for principal oryx/127.11.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.60.keytab.
I20260926 05:47:05.806461 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:36611
--webserver_interface=127.11.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.60.keytab
--principal=oryx/127.11.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:05.899606 18996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:05.899829 18996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:05.899853 18996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:05.900995 18996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:05.901033 18996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:05.901048 18996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:05.901060 18996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:05.901072 18996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:05.902694 18996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.60.keytab
--principal=oryx/127.11.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:05.902920 18996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:05.903108 18996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:05.908509 19004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.908519 19001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:05.908524 19002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:05.908573 18996 server_base.cc:1034] running on GCE node
I20260926 05:47:05.909017 18996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:05.909355 18996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:05.910637 18996 hybrid_clock.cc:648] HybridClock initialized: now 1790401625910599 us; error 56 us; skew 500 ppm
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:05.913512 18996 init.cc:377] Logged in from keytab as oryx/127.11.247.60@KRBTEST.COM (short username oryx)
I20260926 05:47:05.914062 18996 webserver.cc:460] Webserver started at http://127.11.247.60:42363/ using document root <none> and password file <none>
I20260926 05:47:05.914351 18996 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:05.914433 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:05.914681 18996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:05.915920 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/instance:
uuid: "7194a79e54c2417aa0a001e40337d8df"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "0b69241c530ea8580f2189967d0a2c15"
server_key_iv: "62501930b2f7ea7651622ac4d181fc12"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.916357 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal/instance:
uuid: "7194a79e54c2417aa0a001e40337d8df"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "0b69241c530ea8580f2189967d0a2c15"
server_key_iv: "62501930b2f7ea7651622ac4d181fc12"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.918010 18996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:47:05.919047 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:05.919373 18996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:05.919502 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
uuid: "7194a79e54c2417aa0a001e40337d8df"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "0b69241c530ea8580f2189967d0a2c15"
server_key_iv: "62501930b2f7ea7651622ac4d181fc12"
server_key_version: "encryptionkey@0"
I20260926 05:47:05.919591 18996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:05.931991 18996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:05.934491 18996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:05.934726 18996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:05.939317 18996 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.60:36611
I20260926 05:47:05.939375 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.60:36611 every 8 connection(s)
I20260926 05:47:05.939740 18996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
I20260926 05:47:05.940953 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:05.942232 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 18996
I20260926 05:47:05.943090 19064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.60@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.247.60@KRBTEST.COM
I20260926 05:47:05.954447 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.943940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.11.247.62:42167)
0926 05:47:05.944008 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.944011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.944137 (+ 126us) server_negotiation.cc:407] Connection header received
0926 05:47:05.944224 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.944225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.944236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.944324 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.945113 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.945934 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.947373 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.947677 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.947755 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.950719 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.950721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.950721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.950731 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.952474 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.953129 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.953130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.953131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.953217 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.953599 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.953600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.953601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.953759 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.953856 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.954167 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.954315 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:47:05.954603 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.943794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54390)
0926 05:47:05.944026 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.944064 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.944160 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.944351 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.944353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.944740 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.945008 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.945014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.946060 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.946061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.947262 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.947265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.947832 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.947833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.947943 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.948583 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.948594 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.950515 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.952567 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.952590 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.952593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.953050 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.953340 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.953340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.953341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.953484 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.953847 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.953852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.954035 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.954132 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.954328 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260926 05:47:05.954603 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.943163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.11.247.60:36611)
0926 05:47:05.943383 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.943392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.943420 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:47:05.944138 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.944149 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.944177 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.944286 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.945084 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.946268 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.947371 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.947873 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.947942 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.951099 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.951107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.951107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.951115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.953039 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.953459 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.953462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.953462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.953514 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.953721 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.953722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.953722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.954042 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.954184 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.954316 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.954379 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260926 05:47:05.955585 19064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.60@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
I20260926 05:47:05.962549 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.955986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.11.247.61:46189)
0926 05:47:05.956051 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.956053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.956062 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:05.956087 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.956087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.956108 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.956160 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.956464 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.957036 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.957725 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.957929 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.957990 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.959907 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.959909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.959909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.959917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.961503 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.961847 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.961848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.961848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.961897 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.962060 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.962060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.962060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.962163 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.962232 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.962410 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.962479 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:47:05.962724 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.955885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37044)
0926 05:47:05.955941 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.955952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.955994 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.956171 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.956172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.956322 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.956367 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.956369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.957110 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.957111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.957630 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.957632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.958001 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.958002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.958054 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.958450 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.958454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.959744 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.961595 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.961596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.961597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.961775 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.961955 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.961956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.961956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.962011 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.962262 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.962263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.962311 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.962391 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.962550 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:47:05.963209 19064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:47:05 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401625, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.60@KRBTEST.COM for oryx/127.11.247.60@KRBTEST.COM
I20260926 05:47:05.969722 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.963581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.11.247.60:36611)
0926 05:47:05.963637 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.963639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.963646 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:05.963663 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.963663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.963671 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.963733 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.963977 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.964478 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.965125 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.965344 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.965428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.967312 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.967314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.967314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.967321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.968565 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.968936 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.968937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.968937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.968994 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.969184 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.969185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.969186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.969305 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.969382 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.969509 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.969606 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260926 05:47:05.969730 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.963497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41138)
0926 05:47:05.963552 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.963562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.963600 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.963736 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.963737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.963872 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.963904 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.963905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.964572 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.964573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.965045 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.965046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.965427 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.965428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.965476 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.965923 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.965927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.967205 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.968666 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.968667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.968667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.968831 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.969061 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.969062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.969062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.969118 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.969375 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.969375 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.969413 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.969481 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.969651 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":77}
I20260926 05:47:05.971588 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Bootstrap starting.
I20260926 05:47:05.972476 19064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:05.972826 19064 log.cc:824] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:05.973754 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: No bootstrap required, opened a new log
I20260926 05:47:05.975567 19064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } }
I20260926 05:47:05.975744 19064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:05.975801 19064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7194a79e54c2417aa0a001e40337d8df, State: Initialized, Role: FOLLOWER
I20260926 05:47:05.975952 19064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } }
I20260926 05:47:05.976250 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } } }
I20260926 05:47:05.976300 19064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:05.976370 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:47:05.978096 19080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:47:05.978157 19080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:47:05.981459 18996 master_runner.cc:428] Detected that this master 7194a79e54c2417aa0a001e40337d8df is joining an existing cluster
I20260926 05:47:05.981525 18996 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.247.60:36611
I20260926 05:47:05.990082 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.982894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41150)
0926 05:47:05.983120 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.983129 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.983195 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.983359 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.983359 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.983603 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.983687 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.983689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.984597 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.984598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.985456 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.985458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.985921 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.985979 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.986526 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:05.986531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:05.986954 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:05.988915 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.988916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.988916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.989146 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.989431 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:05.989432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:05.989432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:05.989495 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:05.989802 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:05.989803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:05.989842 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 05:47:05.989892 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:47:05.989991 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":63,"threads_started":1}
I20260926 05:47:05.990177 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.983000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.11.247.60:36611)
0926 05:47:05.983059 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.983060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.983173 (+ 113us) server_negotiation.cc:407] Connection header received
0926 05:47:05.983261 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.983262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.983271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.983298 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.983765 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.984514 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985539 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.985818 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985855 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:05.987049 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:05.987050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:05.987050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:05.987059 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:05.988788 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.989248 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.989250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.989250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.989318 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:05.989580 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:05.989581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:05.989582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:05.989738 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:05.989816 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:05.989914 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:47:05.990055 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":39}
I20260926 05:47:06.006315 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18406
I20260926 05:47:06.011787 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:33433
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:42167
--webserver_port=34525
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:47:06.015534 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.982671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.11.247.62:42167)
0926 05:47:05.982725 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.982728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.982868 (+ 140us) server_negotiation.cc:407] Connection header received
0926 05:47:05.982965 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.982967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.982981 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.983046 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.983521 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.984313 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985464 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.985768 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985812 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.012558 (+ 26746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.012560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.012561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.012569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.014174 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.014645 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.014646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.014647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.014708 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.014914 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.014914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.014915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.015060 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.015110 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.015292 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.015428 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:47:06.015918 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.982634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54404)
0926 05:47:05.982791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.982804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.982898 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.983089 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.983090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.983316 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.983417 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.983419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.984428 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.984429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.985360 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.985362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.985910 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.985975 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.011453 (+ 25478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.011461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.011944 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.014275 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.014276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.014276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.014519 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.014784 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.014784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.014784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.014846 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.015141 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.015142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.015187 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.015686 (+ 499us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.015807 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260926 05:47:06.016672 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.983161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.11.247.61:46189)
0926 05:47:05.983205 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:05.983207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:05.983470 (+ 263us) server_negotiation.cc:407] Connection header received
0926 05:47:05.983564 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:05.983565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:05.983576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:05.983606 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:05.984003 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.984747 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.985724 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.986014 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.986055 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.012540 (+ 26485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.012542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.012543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.012553 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.014506 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.015018 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.015019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.015020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.015080 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.015312 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.015313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.015313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.015466 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.015536 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.016424 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.016529 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:47:06.017409 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:05.983261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37060)
0926 05:47:05.983412 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:05.983421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:05.983493 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:05.983681 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:05.983682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:05.983864 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:05.983915 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.983917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.984832 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.984833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.985645 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:05.985646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:05.986130 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:05.986131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:05.986564 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.012064 (+ 25500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.012068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.012423 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.014652 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.014653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.014653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.014920 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.015164 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.015165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.015165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.015237 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.015869 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.015870 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.015937 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.017130 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.017262 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401624, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.62@KRBTEST.COM for oryx/127.11.247.60@KRBTEST.COM
I20260926 05:47:06.027235 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.018424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41162)
0926 05:47:06.018478 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.018489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.018525 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.018790 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.018792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.018987 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.019049 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.019051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.020051 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.020053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.021097 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.021101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.021596 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.021598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.021681 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.022163 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.022169 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.023992 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.025984 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.025986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.025986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.026237 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.026466 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.026467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.026467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.026541 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.026858 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.026859 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.026917 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.026990 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.027113 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:47:06.027235 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.018442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.11.247.60:36611)
0926 05:47:06.018660 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.018662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.018669 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:06.018700 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.018701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.018711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.018762 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.019168 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.019949 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.021200 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.021507 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.021583 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.024094 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.024096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.024097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.024105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.025887 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.026320 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.026321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.026321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.026392 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.026638 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.026639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.026639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.026783 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.026852 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.027029 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.027122 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260926 05:47:06.028264 18344 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.247.60:36611: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7194a79e54c2417aa0a001e40337d8df"
member_type: NON_VOTER
last_known_addr {
host: "127.11.247.60"
port: 36611
}
attrs {
promote: true
}
}
dest_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b"
cas_config_opid_index: 5
I20260926 05:47:06.028510 18344 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } }
I20260926 05:47:06.028892 18903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7194a79e54c2417aa0a001e40337d8df (127.11.247.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.029850 18903 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.247.60:36611
I20260926 05:47:06.029949 18903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.030025 18903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:06.030355 18996 master.cc:537] Master@127.11.247.60:36611 shutting down...
W20260926 05:47:06.031141 18329 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b -> Peer 7194a79e54c2417aa0a001e40337d8df (127.11.247.60:36611): Couldn't send request to peer 7194a79e54c2417aa0a001e40337d8df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:47:06.033519 18996 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:06.033619 18996 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:06.033663 18996 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: stopping tablet replica
I20260926 05:47:06.035472 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.028959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37070)
0926 05:47:06.029134 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.029144 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.029185 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.029416 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.029423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.029425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.029472 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.029473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.030944 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.030944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.033605 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.033611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.035178 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.035179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.035256 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.035286 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.035340 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.035390 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 05:47:06.035737 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.029057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.11.247.61:46189)
0926 05:47:06.029110 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.029112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.029248 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:47:06.029274 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.029275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.029300 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.029356 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.029543 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.030323 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.033744 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.034967 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.035051 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.035134 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.035630 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.035679 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:47:06.036079 18952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 LEARNER]: Refusing update from remote peer a9cf3850e4ed4be1a1462dbb86082e5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 05:47:06.036288 18904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 05:47:06.036870 18952 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7194a79e54c2417aa0a001e40337d8df (127.11.247.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.038340 18904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: attempting to promote NON_VOTER 93e906df8f6849c293efc589a8d96490 to VOTER
I20260926 05:47:06.038336 19091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.038342 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9cf3850e4ed4be1a1462dbb86082e5b. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: NON_VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: true } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.038450 19091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:47:06.038456 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:47:06.038568 18904 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } }
I20260926 05:47:06.039274 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.039121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41178)
0926 05:47:06.039179 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.039240 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:36611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:47:06.039558 18952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 LEARNER]: Refusing update from remote peer a9cf3850e4ed4be1a1462dbb86082e5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 05:47:06.039620 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.039537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41190)
0926 05:47:06.039575 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.039597 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:36611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 05:47:06.039681 18329 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b -> Peer 7194a79e54c2417aa0a001e40337d8df (127.11.247.60:36611): Couldn't send request to peer 7194a79e54c2417aa0a001e40337d8df. Status: Network error: Client connection negotiation failed: client connection to 127.11.247.60:36611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 05:47:06.039862 18903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 05:47:06.040628 18903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93e906df8f6849c293efc589a8d96490 (127.11.247.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.041565 18903 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.041631 18903 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:06.041610 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.041766 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:06.042038 18952 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93e906df8f6849c293efc589a8d96490 (127.11.247.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.042985 18981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.043053 18981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:47:06.047802 18996 master.cc:552] Master@127.11.247.60:36611 shutdown complete.
I20260926 05:47:06.047859 18996 master_runner.cc:305] Clearing existing system tablet
I20260926 05:47:06.047988 18996 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:06.048033 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:06.048805 18996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:06.049360 19094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.049505 18996 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:06.049567 18996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.049590 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
uuid: "7194a79e54c2417aa0a001e40337d8df"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "0b69241c530ea8580f2189967d0a2c15"
server_key_iv: "62501930b2f7ea7651622ac4d181fc12"
server_key_version: "encryptionkey@0"
I20260926 05:47:06.050318 18996 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:06.052011 18996 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 05:47:06.052090 18996 log.cc:1196] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 05:47:06.052301 18996 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Deleting consensus metadata
I20260926 05:47:06.052433 18996 master_runner.cc:315] Copying system tablet from 127.11.247.62:42167
I20260926 05:47:06.053169 18996 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: tablet copy: Beginning tablet copy session from remote peer at address 127.11.247.62:42167
I20260926 05:47:06.061935 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.053595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.11.247.62:42167)
0926 05:47:06.053648 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.053650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.054794 (+ 1144us) server_negotiation.cc:407] Connection header received
0926 05:47:06.054844 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.054845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.054859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.054938 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.055347 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.056122 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.057170 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.057501 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.057578 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.058837 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.058839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.058839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.058847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.060571 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.061017 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.061019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.061019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.061083 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.061359 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.061360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.061360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.061502 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.061576 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.061753 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.061837 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:47:06.061944 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.053507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54416)
0926 05:47:06.054682 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.054696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.054787 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.054953 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.054954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.055186 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.055248 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.055250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.056233 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.056234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.057067 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.057070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.057649 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.057650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.057713 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.058267 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.058273 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.058739 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.060675 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.060695 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.060696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.060934 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.061160 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.061161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.061162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.061295 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.061582 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.061583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.061636 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.061713 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.061833 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"thread_start_us":76,"threads_started":1}
I20260926 05:47:06.062657 18364 tablet_copy_service.cc:140] P a9cf3850e4ed4be1a1462dbb86082e5b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7194a79e54c2417aa0a001e40337d8df ({username='oryx', principal='oryx/127.11.247.60@KRBTEST.COM'} at 127.0.0.1:54416)
I20260926 05:47:06.062736 18364 tablet_copy_service.cc:161] P a9cf3850e4ed4be1a1462dbb86082e5b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7194a79e54c2417aa0a001e40337d8df at {username='oryx', principal='oryx/127.11.247.60@KRBTEST.COM'} at 127.0.0.1:54416: session id = 7194a79e54c2417aa0a001e40337d8df-00000000000000000000000000000000
I20260926 05:47:06.063329 18364 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:47:06.063827 18996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 05:47:06.065364 18996 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: tablet copy: Starting download of 0 data blocks...
I20260926 05:47:06.065554 18996 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: tablet copy: Starting download of 1 WAL segments...
I20260926 05:47:06.066483 18996 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:47:06.067529 18364 tablet_copy_service.cc:342] P a9cf3850e4ed4be1a1462dbb86082e5b: Request end of tablet copy session 7194a79e54c2417aa0a001e40337d8df-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.247.60@KRBTEST.COM'} at 127.0.0.1:54416
I20260926 05:47:06.067608 18364 tablet_copy_service.cc:433] P a9cf3850e4ed4be1a1462dbb86082e5b: ending tablet copy session 7194a79e54c2417aa0a001e40337d8df-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7194a79e54c2417aa0a001e40337d8df
I20260926 05:47:06.070779 18996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:06.072069 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:06.072206 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.072429 18996 server_base.cc:1034] running on GCE node
W20260926 05:47:06.074795 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.074992 18996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:06.075172 18996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:06.076319 18996 hybrid_clock.cc:648] HybridClock initialized: now 1790401626076299 us; error 45 us; skew 500 ppm
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:06.077987 18996 init.cc:377] Logged in from keytab as oryx/127.11.247.60@KRBTEST.COM (short username oryx)
I20260926 05:47:06.078294 18996 webserver.cc:460] Webserver started at http://127.11.247.60:33865/ using document root <none> and password file <none>
I20260926 05:47:06.078419 18996 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:06.078460 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:06.079303 18996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:06.079937 19111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.080114 18996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:47:06.080174 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
uuid: "7194a79e54c2417aa0a001e40337d8df"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "0b69241c530ea8580f2189967d0a2c15"
server_key_iv: "62501930b2f7ea7651622ac4d181fc12"
server_key_version: "encryptionkey@0"
I20260926 05:47:06.080250 18996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:06.090251 18996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:06.092334 18996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:06.095729 18996 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.60:36611
I20260926 05:47:06.096167 18996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
I20260926 05:47:06.097119 19164 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:47:06.097656 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Bootstrap starting.
I20260926 05:47:06.098709 19163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.60:36611 every 8 connection(s)
I20260926 05:47:06.100818 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:47:06.101051 19164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: Bootstrap complete.
I20260926 05:47:06.101263 19164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } }
I20260926 05:47:06.101356 19164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7194a79e54c2417aa0a001e40337d8df, State: Initialized, Role: LEARNER
I20260926 05:47:06.101392 19164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } }
I20260926 05:47:06.101606 19164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:06.101619 19167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.101881 19167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: This master's current role is: LEARNER
I20260926 05:47:06.103219 19178 catalog_manager.cc:1259] Loaded cluster ID: f491556a2f234ef88e8eb1db84ac071e
I20260926 05:47:06.103322 19178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: loading cluster ID for follower catalog manager: success
I20260926 05:47:06.104687 19178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: acquiring CA information for follower catalog manager: success
I20260926 05:47:06.105036 19178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 05:47:06.121930 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:06.122166 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:06.122192 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:06.123612 19089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:06.123672 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:06.123703 19089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:06.125331 19089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:33433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=34525
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:06.125603 19089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:06.125859 19089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:06.132239 19184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:06.132230 19183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.132290 19089 server_base.cc:1034] running on GCE node
W20260926 05:47:06.132437 19186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.132639 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:06.132948 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:06.134128 19089 hybrid_clock.cc:648] HybridClock initialized: now 1790401626134115 us; error 27 us; skew 500 ppm
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:06.137213 19089 init.cc:377] Logged in from keytab as oryx/127.11.247.1@KRBTEST.COM (short username oryx)
I20260926 05:47:06.137807 19089 webserver.cc:460] Webserver started at http://127.11.247.1:34525/ using document root <none> and password file <none>
I20260926 05:47:06.138077 19089 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:06.138161 19089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:06.139813 19089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:06.140605 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.140825 19089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:06.140939 19089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "8e46366f58cb4b23afecfd2e39986d2b"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "5a20bff365ee5454deb29950840ea754"
server_key_iv: "d43a42731df9c06adb7b8e8c788cb3ca"
server_key_version: "encryptionkey@0"
I20260926 05:47:06.141417 19089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:06.162880 19089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:06.165343 19089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:06.165493 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:06.165707 19089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:06.166049 19089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:06.166085 19089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.166109 19089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:06.166155 19089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.172194 19089 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:33433
I20260926 05:47:06.172289 19306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:33433 every 8 connection(s)
I20260926 05:47:06.172657 19089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:06.179518 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 19089
I20260926 05:47:06.179697 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18540
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.60@KRBTEST.COM
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:06.185763 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.173557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:52925 (local address 127.11.247.61:46189)
0926 05:47:06.173683 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.173685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.173981 (+ 296us) server_negotiation.cc:407] Connection header received
0926 05:47:06.175467 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.175468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.175480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.175528 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.176238 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.176736 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.177977 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.178267 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178323 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.182486 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.182488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.182488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.182498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.184185 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184651 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.184652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.184653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.184714 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184921 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.184922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.184922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.185072 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.185117 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.185575 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.185664 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 05:47:06.185976 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.173472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.11.247.1:39191)
0926 05:47:06.173815 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.173877 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.175293 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.175587 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.175588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.175917 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.176141 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.176147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.177353 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.177354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.178106 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.178107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.178632 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.178703 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.179301 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.179306 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.181843 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.184041 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184052 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.184055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.184426 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.184671 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.184672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.184826 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.185154 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.185162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.185386 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.185528 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.185717 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260926 05:47:06.186115 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.173472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.11.247.1:52925)
0926 05:47:06.173879 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.173889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.175288 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.175571 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.175577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.175938 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.176139 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.176144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.176852 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.176853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.177868 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.177870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.178375 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.178453 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.179077 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.179085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.182369 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.184270 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.184271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.184549 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.184788 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.184789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.184848 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.185164 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.185164 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.185386 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.185908 (+ 522us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.186016 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":133,"thread_start_us":63,"threads_started":1}
I20260926 05:47:06.186333 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.173569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:39191 (local address 127.11.247.60:36611)
0926 05:47:06.173737 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.173739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.174043 (+ 304us) server_negotiation.cc:407] Connection header received
0926 05:47:06.175457 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.175459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.175475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.175538 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.176249 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.177268 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178187 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.178508 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178613 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.181958 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.182016 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.182016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.182026 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.183935 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184529 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.184531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.184531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.184599 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.184912 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.184913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.184913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.185065 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.185126 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.186104 (+ 978us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.186210 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260926 05:47:06.186674 19309 heartbeater.cc:347] Connected to a master server at 127.11.247.61:46189
I20260926 05:47:06.186785 19309 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.187024 19309 heartbeater.cc:511] Master 127.11.247.61:46189 requested a full tablet report, sending...
I20260926 05:47:06.187638 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.173472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:42421)
0926 05:47:06.173850 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.173877 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.175297 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.175570 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.175575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.175924 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.176141 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.176147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.177105 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.177106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.177835 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.177837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.178316 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.178452 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.179133 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.179138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.182369 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.185258 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.185260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.185260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.185523 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.185750 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.185751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.185751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.186648 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.187043 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.187044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.187150 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.187223 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.187331 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":14,"thread_start_us":88,"threads_started":1}
I20260926 05:47:06.187709 18942 ts_manager.cc:194] Registered new tserver with Master: 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433)
I20260926 05:47:06.187741 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.173735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:42421 (local address 127.11.247.62:42167)
0926 05:47:06.173786 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.173788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.173957 (+ 169us) server_negotiation.cc:407] Connection header received
0926 05:47:06.175398 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.175399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.175412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.175479 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.176251 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.177015 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.177943 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.178237 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.178304 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.182472 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.182474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.182474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.182483 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.185148 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.185614 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.185616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.185616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.185680 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.186778 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.186779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.186779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.186954 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.187032 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.187492 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.187596 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:47:06.188325 19307 heartbeater.cc:347] Connected to a master server at 127.11.247.60:36611
I20260926 05:47:06.188357 19308 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42167
I20260926 05:47:06.188386 19307 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.188431 19308 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.188532 19307 heartbeater.cc:511] Master 127.11.247.60:36611 requested a full tablet report, sending...
I20260926 05:47:06.188877 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:38785
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:42167
--webserver_port=37201
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:47:06.188937 19128 ts_manager.cc:194] Registered new tserver with Master: 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433)
I20260926 05:47:06.188535 19308 heartbeater.cc:511] Master 127.11.247.62:42167 requested a full tablet report, sending...
I20260926 05:47:06.190534 18344 ts_manager.cc:194] Re-registered known tserver with Master: 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433)
I20260926 05:47:06.191227 18344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:42421
W20260926 05:47:06.275940 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:06.276185 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:06.276211 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:06.277446 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:06.277493 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:06.277529 19315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:06.279217 19315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:38785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=37201
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:06.279744 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:06.280061 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:06.285977 19323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.286064 19315 server_base.cc:1034] running on GCE node
W20260926 05:47:06.285991 19321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:06.286005 19320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.286550 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:06.286906 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:06.288107 19315 hybrid_clock.cc:648] HybridClock initialized: now 1790401626288100 us; error 40 us; skew 500 ppm
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:06.291121 19315 init.cc:377] Logged in from keytab as oryx/127.11.247.2@KRBTEST.COM (short username oryx)
I20260926 05:47:06.291920 19315 webserver.cc:460] Webserver started at http://127.11.247.2:37201/ using document root <none> and password file <none>
I20260926 05:47:06.292209 19315 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:06.292301 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:06.293900 19315 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:06.294941 19330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.295265 19315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:06.295411 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "18e03719a5bb43f8ade3cfb83377062d"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "6714bfad35531f6a2086dfe22564802f"
server_key_iv: "7e7d9885b3bb729a3953723a700606bc"
server_key_version: "encryptionkey@0"
I20260926 05:47:06.295825 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:06.306449 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:06.309211 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:06.309398 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:06.309655 19315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:06.310040 19315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:06.310092 19315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.310149 19315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:06.310177 19315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.316330 19315 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:38785
I20260926 05:47:06.316371 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:38785 every 8 connection(s)
I20260926 05:47:06.316741 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.60@KRBTEST.COM
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
I20260926 05:47:06.325423 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 19315
I20260926 05:47:06.325590 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18674
I20260926 05:47:06.328971 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.317752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:43881 (local address 127.11.247.62:42167)
0926 05:47:06.317799 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.317801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.317920 (+ 119us) server_negotiation.cc:407] Connection header received
0926 05:47:06.318718 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.318720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.318733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.318764 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.319601 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.320117 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321195 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.321401 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321481 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.324135 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.324136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.324137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.324147 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.325990 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.328001 (+ 2011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.328003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.328003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.328073 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.328284 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.328285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.328285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.328414 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.328497 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.328702 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.328798 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:47:06.329062 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.317529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:36513 (local address 127.11.247.61:46189)
0926 05:47:06.317579 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.317581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.317909 (+ 328us) server_negotiation.cc:407] Connection header received
0926 05:47:06.318648 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.318650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.318665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.318699 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.319613 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.320393 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321532 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.321936 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321979 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.325046 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.325048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.325048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.325058 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.326938 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.327599 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.327601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.327601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.327679 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.328010 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.328012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.328012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.328197 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.328272 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.328704 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.328823 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:47:06.329155 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.317671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:43881)
0926 05:47:06.317847 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.317859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.318505 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.318854 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.318855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.319265 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.319494 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.319499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.320232 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.320233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.321086 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.321089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.321486 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.321625 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.322187 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.322196 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.323983 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.327589 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.327590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.327590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.327876 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.328140 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.328140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.328141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.328183 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.328488 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.328489 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.328543 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.328669 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.328872 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260926 05:47:06.329633 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.317544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.11.247.2:36513)
0926 05:47:06.317748 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.317812 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.318455 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.318790 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.318795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.319303 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.319496 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.319503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.320501 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.320501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.321428 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.321430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.322051 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.322051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.322109 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.322562 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.322566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.324943 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.327046 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.327061 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.327065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.327500 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.327760 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.327761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.327761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.327929 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.328305 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.328312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.328528 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.329221 (+ 693us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.329373 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":60,"thread_start_us":91,"threads_started":1}
I20260926 05:47:06.329677 19445 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42167
I20260926 05:47:06.329771 19445 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.329972 19445 heartbeater.cc:511] Master 127.11.247.62:42167 requested a full tablet report, sending...
I20260926 05:47:06.330042 19446 heartbeater.cc:347] Connected to a master server at 127.11.247.61:46189
I20260926 05:47:06.330088 19446 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.330168 19446 heartbeater.cc:511] Master 127.11.247.61:46189 requested a full tablet report, sending...
I20260926 05:47:06.330354 18344 ts_manager.cc:194] Re-registered known tserver with Master: 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:06.330952 18344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:43881
I20260926 05:47:06.331281 18942 ts_manager.cc:194] Registered new tserver with Master: 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:06.332239 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:40379
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:42167
--webserver_port=36979
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:47:06.333366 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.317594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:41989 (local address 127.11.247.60:36611)
0926 05:47:06.317634 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.317636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.317960 (+ 324us) server_negotiation.cc:407] Connection header received
0926 05:47:06.318622 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.318624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.318642 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.318703 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.319615 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.320398 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321533 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.321846 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321889 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.324607 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.324609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.324609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.324619 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.327489 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.332404 (+ 4915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.332407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.332407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.332509 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.332766 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.332767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.332767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.332904 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.332963 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.333159 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.333300 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:47:06.333747 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.317544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.11.247.2:41989)
0926 05:47:06.317898 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.317908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.318455 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.318790 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.318795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.319301 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.319505 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.319507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.320498 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.320498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.321428 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.321430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.321976 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.321977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.322047 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.322682 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.322686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.324504 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.332017 (+ 7513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.332019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.332019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.332292 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.332600 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.332601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.332602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.332677 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.332974 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.332975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.333024 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.333100 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.333241 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":309,"thread_start_us":63,"threads_started":1}
I20260926 05:47:06.334185 19444 heartbeater.cc:347] Connected to a master server at 127.11.247.60:36611
I20260926 05:47:06.334283 19444 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.334383 19444 heartbeater.cc:511] Master 127.11.247.60:36611 requested a full tablet report, sending...
I20260926 05:47:06.334753 19128 ts_manager.cc:194] Registered new tserver with Master: 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
W20260926 05:47:06.421522 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:06.421763 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:06.421789 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:06.422999 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:06.423044 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:06.423074 19451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:06.424589 19451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:40379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=36979
--tserver_master_addrs=127.11.247.62:42167,127.11.247.61:46189,127.11.247.60:36611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:06.424806 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:06.425009 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:06.430568 19459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:06.430568 19456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:06.430651 19457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:06.430676 19451 server_base.cc:1034] running on GCE node
I20260926 05:47:06.431087 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:06.431318 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:06.432497 19451 hybrid_clock.cc:648] HybridClock initialized: now 1790401626432481 us; error 48 us; skew 500 ppm
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:06.435287 19451 init.cc:377] Logged in from keytab as oryx/127.11.247.3@KRBTEST.COM (short username oryx)
I20260926 05:47:06.435797 19451 webserver.cc:460] Webserver started at http://127.11.247.3:36979/ using document root <none> and password file <none>
I20260926 05:47:06.436003 19451 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:06.436060 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:06.437456 19451 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:47:06.438122 19466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.438301 19451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:47:06.438374 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "1589b65ad8544232b5c76d9cb9d44c64"
format_stamp: "Formatted at 2026-09-26 05:47:05 on dist-test-slave-21p4"
server_key: "dfdb04af82a3dfa22efa2a4c47aaf7bf"
server_key_iv: "1436a46ccb876abde4a1b0810137c9a7"
server_key_version: "encryptionkey@0"
I20260926 05:47:06.438804 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:06.471242 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:06.473878 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:06.474016 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:06.474357 19451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:06.474803 19451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:06.474864 19451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.474907 19451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:06.474933 19451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:06.481339 19451 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:40379
I20260926 05:47:06.481458 19579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:40379 every 8 connection(s)
I20260926 05:47:06.481825 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:06.489033 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 19451
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.61@KRBTEST.COM
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.60@KRBTEST.COM
Sep 26 05:47:06 dist-test-slave-21p4 krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401626, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:06.495036 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.483214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:56699 (local address 127.11.247.61:46189)
0926 05:47:06.483290 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.483292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.484432 (+ 1140us) server_negotiation.cc:407] Connection header received
0926 05:47:06.484494 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.484496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.484508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.484562 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.486007 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.486780 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487710 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.488003 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.488047 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.491315 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.491317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.491317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.491325 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.493209 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.493778 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.493780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.493780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.493846 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494151 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.494151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.494152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.494289 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.494340 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.494829 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.494916 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:47:06.495414 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.482684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:52453 (local address 127.11.247.62:42167)
0926 05:47:06.482760 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.482762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.483332 (+ 570us) server_negotiation.cc:407] Connection header received
0926 05:47:06.484364 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.484366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.484381 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.484437 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.485351 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.485877 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487265 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.487551 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487617 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.492342 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.492344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.492345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.492354 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.494000 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494554 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.494556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.494556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.494646 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494871 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.494872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.494873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.495004 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.495055 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.495237 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.495319 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:47:06.495680 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.482672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.11.247.3:35435)
0926 05:47:06.483277 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.483286 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.484331 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.484642 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.484642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.485056 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.485243 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.485250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.485992 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.485993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.487209 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.487212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.487708 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.487824 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.488505 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.488511 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.491760 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.494124 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.494125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.494415 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.494668 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.494669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.494726 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.495015 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.495016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.495056 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.495127 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.495347 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":146,"thread_start_us":272,"threads_started":1}
I20260926 05:47:06.495774 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.482672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:52453)
0926 05:47:06.483152 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.483197 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.484198 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.484582 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.484586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.485020 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.485243 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.485253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.485993 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.485995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.487168 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.487170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.487643 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.487824 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.488454 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.488463 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.492259 (+ 3796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.494189 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.494191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.494466 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.494720 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.494721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.494797 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.495087 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.495088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.495130 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.495505 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.495619 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":25,"thread_start_us":303,"threads_started":1}
I20260926 05:47:06.496179 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.482813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:35435 (local address 127.11.247.60:36611)
0926 05:47:06.482854 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.482856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.483352 (+ 496us) server_negotiation.cc:407] Connection header received
0926 05:47:06.484489 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.484490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.484506 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.484588 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.485377 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.485876 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487335 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.487645 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.487710 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.491864 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.491866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.491866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.491874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.494040 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494489 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.494490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.494490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.494552 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.494793 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.494794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.494794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.494928 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.494988 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.495983 (+ 995us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.496080 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:47:06.496285 19581 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42167
I20260926 05:47:06.496381 19581 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.496582 19581 heartbeater.cc:511] Master 127.11.247.62:42167 requested a full tablet report, sending...
I20260926 05:47:06.497062 18344 ts_manager.cc:194] Re-registered known tserver with Master: 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3:40379)
I20260926 05:47:06.497437 18344 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:52453
I20260926 05:47:06.497668 19580 heartbeater.cc:347] Connected to a master server at 127.11.247.60:36611
I20260926 05:47:06.497730 19580 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.497802 19580 heartbeater.cc:511] Master 127.11.247.60:36611 requested a full tablet report, sending...
I20260926 05:47:06.498096 19128 ts_manager.cc:194] Registered new tserver with Master: 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3:40379)
I20260926 05:47:06.498530 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.483116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.11.247.3:56699)
0926 05:47:06.484350 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.484358 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.484433 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.484584 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.484591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.485024 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:06.485243 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.485250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.486874 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.486875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.487609 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.487611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.488132 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.488133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.488183 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.488673 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:06.488677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:06.491189 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:06.493303 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.493313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.493316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.493662 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.493924 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:06.493925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:06.493925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:06.494072 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:06.494391 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:06.494399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:06.494645 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.496620 (+ 1975us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.497923 (+ 1303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"mutex_wait_us":9,"thread_start_us":83,"threads_started":1}
I20260926 05:47:06.498894 19582 heartbeater.cc:347] Connected to a master server at 127.11.247.61:46189
I20260926 05:47:06.498947 19582 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:06.499022 19582 heartbeater.cc:511] Master 127.11.247.61:46189 requested a full tablet report, sending...
I20260926 05:47:06.499330 18942 ts_manager.cc:194] Registered new tserver with Master: 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3:40379)
I20260926 05:47:06.501688 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.490482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.11.247.61:46189)
0926 05:47:06.490657 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.490659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.490666 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:06.490687 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.490687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.490700 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.491198 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.492574 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.493385 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.496087 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.496395 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.496712 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.498015 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.498016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.498017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.498024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.500347 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.500718 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.500720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.500720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.500781 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.501036 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.501038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.501038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.501171 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.501304 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.501468 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.501567 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":54,"threads_started":1}
I20260926 05:47:06.502079 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.490173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.11.247.60:36611)
0926 05:47:06.490456 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.490458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.490643 (+ 185us) server_negotiation.cc:407] Connection header received
0926 05:47:06.490671 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.490673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.490685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.490739 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.491546 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.494112 (+ 2566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.495601 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.495919 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.496009 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.498804 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.498806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.498806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.498815 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.500550 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.501005 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.501007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.501007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.501069 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.501285 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.501285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.501286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.501481 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.501587 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.501831 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.501925 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":58,"threads_started":1}
I20260926 05:47:06.505684 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.492424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.11.247.62:42167)
0926 05:47:06.495489 (+ 3065us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.495491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.497014 (+ 1523us) server_negotiation.cc:407] Connection header received
0926 05:47:06.498098 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.498100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.498111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.498139 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.498848 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.499626 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.500740 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.501070 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.501114 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.502923 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.502925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.502925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.502932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.504444 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.504971 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.504972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.504973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.505029 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.505240 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.505241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.505241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.505378 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.505458 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.505546 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.505610 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3050,"thread_start_us":106,"threads_started":1}
I20260926 05:47:06.507587 18344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:47:06.509264 18344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:06.517915 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.513761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33433 (local address 127.0.0.1:43556)
0926 05:47:06.513800 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.513813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.513857 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.514467 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.514468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.514469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.514516 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.514517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.515844 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.515845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.516879 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.516881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.517644 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.517644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.517710 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.517714 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.517773 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.517831 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 05:47:06.517927 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.513919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.11.247.1:33433)
0926 05:47:06.514245 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.514250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.514262 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:47:06.514310 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.514313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.514341 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.514458 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.514574 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.515756 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.516976 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.517552 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.517644 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.517736 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.517790 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.517831 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":71,"threads_started":1}
I20260926 05:47:06.518129 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.513802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40379 (local address 127.0.0.1:34618)
0926 05:47:06.513950 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.513960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.513997 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.514804 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.514805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.514807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.514842 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.514842 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.516006 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.516007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.516979 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.516982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.517875 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.517876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.517952 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.517957 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.517991 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.518039 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260926 05:47:06.518386 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.514117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.11.247.2:38785)
0926 05:47:06.514288 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.514292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.514301 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:06.514336 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.514337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.514351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.514458 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.514772 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.515527 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.516988 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.517764 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.518086 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.518174 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.518212 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.518273 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260926 05:47:06.518662 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.513883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:38785 (local address 127.0.0.1:34480)
0926 05:47:06.514080 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.514089 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.514127 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.514645 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.514646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.514648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.514683 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.514686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.515619 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.515620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.516879 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.516881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.517873 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.517875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.517954 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.517959 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.517991 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.518039 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260926 05:47:06.519250 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.514059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.11.247.3:40379)
0926 05:47:06.514627 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.514632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.514643 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:06.514682 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.514685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.514700 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.514807 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.514899 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.515912 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.517053 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.517760 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.518690 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.519101 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.519129 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.519186 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":86,"threads_started":1}
I20260926 05:47:06.519351 19241 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6fe860ec3467d97e6627cd28fbc3d (DEFAULT_TABLE table=test-table [id=d4c5bb978f884ead8091903d05e44d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:06.519556 19378 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6fe860ec3467d97e6627cd28fbc3d (DEFAULT_TABLE table=test-table [id=d4c5bb978f884ead8091903d05e44d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:06.519795 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6fe860ec3467d97e6627cd28fbc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:06.519843 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6fe860ec3467d97e6627cd28fbc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:06.519857 19514 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6fe860ec3467d97e6627cd28fbc3d (DEFAULT_TABLE table=test-table [id=d4c5bb978f884ead8091903d05e44d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:06.520110 19514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6fe860ec3467d97e6627cd28fbc3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:06.523008 19606 tablet_bootstrap.cc:492] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Bootstrap starting.
I20260926 05:47:06.523566 19607 tablet_bootstrap.cc:492] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Bootstrap starting.
I20260926 05:47:06.523864 19606 tablet_bootstrap.cc:654] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:06.524207 19606 log.cc:824] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:06.524237 19607 tablet_bootstrap.cc:654] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:06.524580 19607 log.cc:824] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:06.524633 19608 tablet_bootstrap.cc:492] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Bootstrap starting.
I20260926 05:47:06.525183 19606 tablet_bootstrap.cc:492] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: No bootstrap required, opened a new log
I20260926 05:47:06.525279 19606 ts_tablet_manager.cc:1397] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:06.525279 19608 tablet_bootstrap.cc:654] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:06.525391 19607 tablet_bootstrap.cc:492] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: No bootstrap required, opened a new log
I20260926 05:47:06.525475 19607 ts_tablet_manager.cc:1397] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:47:06.525607 19608 log.cc:824] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:06.526445 19608 tablet_bootstrap.cc:492] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: No bootstrap required, opened a new log
I20260926 05:47:06.526527 19608 ts_tablet_manager.cc:1397] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:47:06.527446 19606 raft_consensus.cc:348] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.527624 19606 raft_consensus.cc:374] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:06.527671 19606 raft_consensus.cc:729] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1589b65ad8544232b5c76d9cb9d44c64, State: Initialized, Role: FOLLOWER
I20260926 05:47:06.527797 19606 consensus_queue.cc:260] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.528117 19606 ts_tablet_manager.cc:1428] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:47:06.528262 19581 heartbeater.cc:503] Master 127.11.247.62:42167 was elected leader, sending a full tablet report...
I20260926 05:47:06.528263 19607 raft_consensus.cc:348] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.528431 19607 raft_consensus.cc:374] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:06.528465 19607 raft_consensus.cc:729] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e46366f58cb4b23afecfd2e39986d2b, State: Initialized, Role: FOLLOWER
I20260926 05:47:06.528611 19607 consensus_queue.cc:260] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.528944 19607 ts_tablet_manager.cc:1428] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:47:06.529490 19608 raft_consensus.cc:348] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.529662 19608 raft_consensus.cc:374] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:06.529695 19608 raft_consensus.cc:729] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e03719a5bb43f8ade3cfb83377062d, State: Initialized, Role: FOLLOWER
I20260926 05:47:06.529806 19608 consensus_queue.cc:260] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.530040 19308 heartbeater.cc:503] Master 127.11.247.62:42167 was elected leader, sending a full tablet report...
I20260926 05:47:06.530099 19608 ts_tablet_manager.cc:1428] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:47:06.530519 19445 heartbeater.cc:503] Master 127.11.247.62:42167 was elected leader, sending a full tablet report...
W20260926 05:47:06.568219 19447 tablet.cc:2369] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:06.649420 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.646369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41218)
0926 05:47:06.646465 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.646480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.646548 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.646842 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.646843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.646846 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.646917 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.646919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.647670 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.647672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.648469 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.648472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.649181 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.649182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.649236 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.649241 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.649300 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.649350 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 05:47:06.649457 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.646537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.11.247.60:36611)
0926 05:47:06.646656 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.646658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.646671 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:47:06.646710 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.646711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.646736 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.646818 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.647042 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.647562 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.648579 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.649081 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.649150 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.649215 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.649355 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.649401 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 05:47:06.649992 19138 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 05:47:06.651391 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9cf3850e4ed4be1a1462dbb86082e5b. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.651516 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: This master's current role is: LEARNER
W20260926 05:47:06.674335 19310 tablet.cc:2369] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:06.694337 19612 raft_consensus.cc:482] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:06.694500 19612 raft_consensus.cc:504] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.695092 19612 leader_election.cc:288] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433), 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:06.698197 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.695498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:38785 (local address 127.11.247.3:36607)
0926 05:47:06.695550 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.695560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.695595 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.695756 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.695757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.695759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.695855 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.695857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.696495 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.696496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.697345 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.697348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.697914 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.697915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.697999 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.698002 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.698077 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.698121 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":42}
I20260926 05:47:06.698195 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.695589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:36607 (local address 127.11.247.2:38785)
0926 05:47:06.695640 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.695642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.695650 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:06.695682 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.695683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.695691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.695755 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.695937 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.696409 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.697450 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.697834 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.697918 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.697928 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.698085 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.698128 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:47:06.698307 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.695341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33433 (local address 127.11.247.3:45879)
0926 05:47:06.695421 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:06.695448 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:06.695518 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:06.695707 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:06.695708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:06.695710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:06.695807 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.695810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.696484 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.696485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.697347 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:06.697350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.698028 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.698029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:06.698116 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.698119 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:06.698191 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:47:06.698235 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":56}
I20260926 05:47:06.698318 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.695417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45879 (local address 127.11.247.1:33433)
0926 05:47:06.695481 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.695483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.695500 (+ 17us) server_negotiation.cc:407] Connection header received
0926 05:47:06.695604 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.695606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.695617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.695702 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:06.695894 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.696383 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.697445 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.697855 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.698029 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.698041 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.698217 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.698258 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:47:06.698616 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6fe860ec3467d97e6627cd28fbc3d" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e03719a5bb43f8ade3cfb83377062d" is_pre_election: true
I20260926 05:47:06.698660 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6fe860ec3467d97e6627cd28fbc3d" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e46366f58cb4b23afecfd2e39986d2b" is_pre_election: true
I20260926 05:47:06.698796 19261 raft_consensus.cc:2454] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 0.
I20260926 05:47:06.698796 19398 raft_consensus.cc:2454] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 0.
I20260926 05:47:06.699028 19467 leader_election.cc:302] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1589b65ad8544232b5c76d9cb9d44c64, 18e03719a5bb43f8ade3cfb83377062d; no voters:
I20260926 05:47:06.699278 19612 raft_consensus.cc:2781] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:06.699373 19612 raft_consensus.cc:482] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:06.699414 19612 raft_consensus.cc:3037] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:06.700176 19612 raft_consensus.cc:504] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.700336 19612 leader_election.cc:288] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 election: Requested vote from peers 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433), 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:06.700562 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6fe860ec3467d97e6627cd28fbc3d" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e03719a5bb43f8ade3cfb83377062d"
I20260926 05:47:06.700551 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6fe860ec3467d97e6627cd28fbc3d" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e46366f58cb4b23afecfd2e39986d2b"
I20260926 05:47:06.700644 19398 raft_consensus.cc:3037] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:06.700644 19261 raft_consensus.cc:3037] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:06.701627 19261 raft_consensus.cc:2454] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 1.
I20260926 05:47:06.701627 19398 raft_consensus.cc:2454] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 1.
I20260926 05:47:06.701782 19467 leader_election.cc:302] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1589b65ad8544232b5c76d9cb9d44c64, 8e46366f58cb4b23afecfd2e39986d2b; no voters:
I20260926 05:47:06.701876 19612 raft_consensus.cc:2781] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:06.702059 19612 raft_consensus.cc:686] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 LEADER]: Becoming Leader. State: Replica: 1589b65ad8544232b5c76d9cb9d44c64, State: Running, Role: LEADER
I20260926 05:47:06.702178 19612 consensus_queue.cc:237] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:06.703168 18343 catalog_manager.cc:5495] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "1589b65ad8544232b5c76d9cb9d44c64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } health_report { overall_health: UNKNOWN } } }
I20260926 05:47:06.703876 18904 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: NON_VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: true } } }
I20260926 05:47:06.703971 18904 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:06.704798 18904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: attempting to promote NON_VOTER 7194a79e54c2417aa0a001e40337d8df to VOTER
I20260926 05:47:06.704943 18904 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:06.705515 19138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 LEARNER]: Refusing update from remote peer a9cf3850e4ed4be1a1462dbb86082e5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 05:47:06.705730 18951 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Refusing update from remote peer a9cf3850e4ed4be1a1462dbb86082e5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 05:47:06.705773 19621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260926 05:47:06.705969 19621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260926 05:47:06.706539 19092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7194a79e54c2417aa0a001e40337d8df (127.11.247.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.706763 18951 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7194a79e54c2417aa0a001e40337d8df (127.11.247.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.706753 19138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7194a79e54c2417aa0a001e40337d8df (127.11.247.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.707477 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.707557 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:06.707616 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.707671 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:06.707832 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.707917 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:47:06.707958 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.708051 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:47:06.707653 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:06.708250 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b [sys.catalog]: This master's current role is: LEADER
W20260926 05:47:06.732517 19583 tablet.cc:2369] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:06.752175 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.749600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.11.247.3:40379)
0926 05:47:06.749669 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.749671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.749680 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:06.749728 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.749729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.749747 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.749887 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:06.750029 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.750547 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.751385 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.751582 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.751661 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.751838 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:06.751943 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:06.752040 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.752058 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.752079 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:47:06.754034 19261 raft_consensus.cc:1261] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 1 FOLLOWER]: Refusing update from remote peer 1589b65ad8544232b5c76d9cb9d44c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:06.754034 19398 raft_consensus.cc:1261] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Refusing update from remote peer 1589b65ad8544232b5c76d9cb9d44c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:06.754420 19619 consensus_queue.cc:1035] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:06.754596 19612 consensus_queue.cc:1035] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:06.757987 19626 mvcc.cc:205] Tried to move back new op lower bound from 7333485063181692928 to 7333485062972997632. Current Snapshot: MvccSnapshot[applied={T|T < 7333485063181692928}]
I20260926 05:47:06.758615 19625 mvcc.cc:205] Tried to move back new op lower bound from 7333485063181692928 to 7333485062972997632. Current Snapshot: MvccSnapshot[applied={T|T < 7333485063181692928}]
I20260926 05:47:06.761171 18343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:06.761282 18343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:06.762759 18343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9cf3850e4ed4be1a1462dbb86082e5b: Sending DeleteTablet for 3 replicas of tablet b3e6fe860ec3467d97e6627cd28fbc3d
I20260926 05:47:06.763283 19241 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6fe860ec3467d97e6627cd28fbc3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:06 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:43556
I20260926 05:47:06.763281 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6fe860ec3467d97e6627cd28fbc3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:06 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:34480
I20260926 05:47:06.763511 19636 tablet_replica.cc:330] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: stopping tablet replica
I20260926 05:47:06.763612 19636 raft_consensus.cc:2229] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:06.763720 19636 raft_consensus.cc:2258] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:06.763981 19637 tablet_replica.cc:330] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: stopping tablet replica
I20260926 05:47:06.764101 19637 raft_consensus.cc:2229] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:06.764245 19637 raft_consensus.cc:2258] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:06.764628 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18312
I20260926 05:47:06.764742 19637 ts_tablet_manager.cc:1905] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:06.766839 19637 ts_tablet_manager.cc:1918] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:06.766985 19637 log.cc:1196] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/b3e6fe860ec3467d97e6627cd28fbc3d
I20260926 05:47:06.767309 19637 ts_tablet_manager.cc:1939] T b3e6fe860ec3467d97e6627cd28fbc3d P 8e46366f58cb4b23afecfd2e39986d2b: Deleting consensus metadata
I20260926 05:47:06.770810 19514 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6fe860ec3467d97e6627cd28fbc3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:06 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:34618
I20260926 05:47:06.771211 19636 ts_tablet_manager.cc:1905] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:47:06.771430 19196 connection.cc:535] server connection from 127.0.0.1:43556 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:47:06.773208 19636 ts_tablet_manager.cc:1918] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:06.773377 19636 log.cc:1196] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/b3e6fe860ec3467d97e6627cd28fbc3d
I20260926 05:47:06.773757 19636 ts_tablet_manager.cc:1939] T b3e6fe860ec3467d97e6627cd28fbc3d P 18e03719a5bb43f8ade3cfb83377062d: Deleting consensus metadata
W20260926 05:47:06.774243 19332 connection.cc:441] server connection from 127.0.0.1:34480 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34480 (request call id 1) could send its response
I20260926 05:47:06.778714 19639 tablet_replica.cc:330] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: stopping tablet replica
I20260926 05:47:06.778865 19639 raft_consensus.cc:2229] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:06.779014 19639 raft_consensus.cc:2258] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:06.779569 19639 ts_tablet_manager.cc:1905] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:06.781347 19639 ts_tablet_manager.cc:1918] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:06.781430 19639 log.cc:1196] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/b3e6fe860ec3467d97e6627cd28fbc3d
I20260926 05:47:06.781709 19639 ts_tablet_manager.cc:1939] T b3e6fe860ec3467d97e6627cd28fbc3d P 1589b65ad8544232b5c76d9cb9d44c64: Deleting consensus metadata
I20260926 05:47:06.782943 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.763899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.11.247.60:36611)
0926 05:47:06.763942 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.763944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.774301 (+ 10357us) server_negotiation.cc:407] Connection header received
0926 05:47:06.774456 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.774457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.774470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.774503 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.775362 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.776158 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.777170 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.777485 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.777528 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.779542 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.779543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.779544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.779551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.781414 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.781931 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.781933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.781933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.781995 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.782232 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.782233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.782233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.782369 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.782468 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.782720 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.782817 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:47:06.785729 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.773336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.11.247.61:46189)
0926 05:47:06.773386 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.773388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.774186 (+ 798us) server_negotiation.cc:407] Connection header received
0926 05:47:06.774456 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.774457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.774473 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.774503 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.775832 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.776551 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.778492 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.778862 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.778911 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.782956 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.782958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.782958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.782968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.784489 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.784821 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.784823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.784823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.784884 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.785075 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.785075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.785076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.785207 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.785284 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.785503 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.785603 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:47:06.787005 19467 connection.cc:441] server connection from 127.0.0.1:34618 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34618 (request call id 1) could send its response
I20260926 05:47:06.789827 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:06.773894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.11.247.60:36611)
0926 05:47:06.773936 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:06.773938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:06.777556 (+ 3618us) server_negotiation.cc:407] Connection header received
0926 05:47:06.777649 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:06.777650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:06.777660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:06.777689 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:06.779455 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.780828 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.784844 (+ 4016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:06.785155 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:06.785195 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:06.787155 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:06.787156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:06.787156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:06.787163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:06.788565 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.789007 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.789008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.789008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.789065 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:06.789291 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:06.789292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:06.789292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:06.789417 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:06.789519 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:06.789631 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 05:47:06.789707 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:47:07.759320 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:07.759004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:50219)
0926 05:47:07.759165 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:07.759262 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":58,"threads_started":1}
W20260926 05:47:07.759718 19581 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:07.759799 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:07.759539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:60185)
0926 05:47:07.759670 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:07.759743 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":48,"threads_started":1}
W20260926 05:47:07.760057 19308 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:07.761358 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:07.761143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:41795)
0926 05:47:07.761240 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:07.761311 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
W20260926 05:47:07.761534 19445 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.263170 19650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9cf3850e4ed4be1a1462dbb86082e5b)
I20260926 05:47:08.263405 19650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:08.263708 19650 leader_election.cc:288] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167), 7194a79e54c2417aa0a001e40337d8df (127.11.247.60:36611)
I20260926 05:47:08.263983 19651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9cf3850e4ed4be1a1462dbb86082e5b)
I20260926 05:47:08.264115 19651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:08.264192 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.263891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54924)
0926 05:47:08.264013 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.264100 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
W20260926 05:47:08.264343 18928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.264667 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.264453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54934)
0926 05:47:08.264551 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.264605 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260926 05:47:08.264695 19651 leader_election.cc:288] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167), 93e906df8f6849c293efc589a8d96490 (127.11.247.61:46189)
W20260926 05:47:08.264874 19114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.264997 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.264900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54944)
0926 05:47:08.264936 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.264965 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:47:08.265125 18928 leader_election.cc:334] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.265409 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.265225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54954)
0926 05:47:08.265338 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.265377 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
W20260926 05:47:08.265486 19114 leader_election.cc:334] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.267534 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.263898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:36611 (local address 127.0.0.1:41250)
0926 05:47:08.264203 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.264214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:08.264284 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:08.264505 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:08.264506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:08.264511 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:08.264573 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:08.264575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.265825 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.265826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:08.266668 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:08.266671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.267244 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.267245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:08.267309 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:08.267313 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:08.267394 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:47:08.267439 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":52,"threads_started":1}
I20260926 05:47:08.267534 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.264038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.11.247.60:36611)
0926 05:47:08.264140 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:08.264143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:08.264344 (+ 201us) server_negotiation.cc:407] Connection header received
0926 05:47:08.264379 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:08.264380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:08.264393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:08.264448 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:08.264653 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.265737 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.266769 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.267144 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.267240 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:08.267252 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:08.267404 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 05:47:08.267447 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":44,"threads_started":1}
I20260926 05:47:08.267536 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.264583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.11.247.61:46189)
0926 05:47:08.264766 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:08.264768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:08.264775 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:08.264796 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:08.264796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:08.264807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:08.264883 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:08.265064 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.265783 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.266780 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.267159 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.267246 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:08.267257 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:08.267408 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:47:08.267451 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260926 05:47:08.267536 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.264467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:46189 (local address 127.0.0.1:37104)
0926 05:47:08.264590 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.264603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:08.264657 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:08.264908 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:08.264910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:08.264912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:08.264968 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:08.264970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.265879 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.265880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:08.266695 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:08.266697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:08.267246 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:08.267247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:08.267316 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:08.267319 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:08.267400 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:47:08.267451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":38,"threads_started":1}
I20260926 05:47:08.267987 19138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e906df8f6849c293efc589a8d96490" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7194a79e54c2417aa0a001e40337d8df" is_pre_election: true
I20260926 05:47:08.268151 19138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e906df8f6849c293efc589a8d96490 in term 1.
I20260926 05:47:08.268157 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7194a79e54c2417aa0a001e40337d8df" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93e906df8f6849c293efc589a8d96490" is_pre_election: true
I20260926 05:47:08.268244 18951 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7194a79e54c2417aa0a001e40337d8df in term 1.
I20260926 05:47:08.268420 19115 leader_election.cc:302] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7194a79e54c2417aa0a001e40337d8df, 93e906df8f6849c293efc589a8d96490; no voters: a9cf3850e4ed4be1a1462dbb86082e5b
I20260926 05:47:08.268435 18927 leader_election.cc:302] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7194a79e54c2417aa0a001e40337d8df, 93e906df8f6849c293efc589a8d96490; no voters: a9cf3850e4ed4be1a1462dbb86082e5b
I20260926 05:47:08.268563 19650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 05:47:08.268610 19650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a9cf3850e4ed4be1a1462dbb86082e5b)
I20260926 05:47:08.268632 19650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:47:08.268625 19651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 05:47:08.268703 19651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Starting leader election (detected failure of leader a9cf3850e4ed4be1a1462dbb86082e5b)
I20260926 05:47:08.268741 19651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:47:08.269577 19650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:08.269577 19651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:08.269737 19650 leader_election.cc:288] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 2 election: Requested vote from peers a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167), 7194a79e54c2417aa0a001e40337d8df (127.11.247.60:36611)
I20260926 05:47:08.269902 19138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e906df8f6849c293efc589a8d96490" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7194a79e54c2417aa0a001e40337d8df"
I20260926 05:47:08.269971 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.269868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54956)
0926 05:47:08.269914 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.269942 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 05:47:08.269985 19651 leader_election.cc:288] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [CANDIDATE]: Term 2 election: Requested vote from peers a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167), 93e906df8f6849c293efc589a8d96490 (127.11.247.61:46189)
I20260926 05:47:08.269996 19138 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93e906df8f6849c293efc589a8d96490 in current term 2: Already voted for candidate 7194a79e54c2417aa0a001e40337d8df in this term.
I20260926 05:47:08.270138 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.270002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54964)
0926 05:47:08.270083 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.270111 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 05:47:08.270103 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7194a79e54c2417aa0a001e40337d8df" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93e906df8f6849c293efc589a8d96490"
I20260926 05:47:08.270200 18951 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7194a79e54c2417aa0a001e40337d8df in current term 2: Already voted for candidate 93e906df8f6849c293efc589a8d96490 in this term.
I20260926 05:47:08.270243 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.270159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54974)
0926 05:47:08.270200 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.270222 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:47:08.270301 18928 leader_election.cc:334] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.270339 18928 leader_election.cc:302] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93e906df8f6849c293efc589a8d96490; no voters: 7194a79e54c2417aa0a001e40337d8df, a9cf3850e4ed4be1a1462dbb86082e5b
I20260926 05:47:08.270443 19650 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 05:47:08.270460 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.270345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:54990)
0926 05:47:08.270405 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.270437 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:47:08.270505 19114 leader_election.cc:334] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:08.270538 19114 leader_election.cc:302] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7194a79e54c2417aa0a001e40337d8df; no voters: 93e906df8f6849c293efc589a8d96490, a9cf3850e4ed4be1a1462dbb86082e5b
I20260926 05:47:08.270678 19651 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 05:47:08.760540 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.760086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:50787)
0926 05:47:08.760427 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.760478 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":110,"threads_started":1}
I20260926 05:47:08.760602 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.760381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:44181)
0926 05:47:08.760515 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.760551 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 05:47:08.762131 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:08.761892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:45117)
0926 05:47:08.762010 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:08.762065 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260926 05:47:09.761281 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:09.761017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:53759)
0926 05:47:09.761168 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:09.761216 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260926 05:47:09.761283 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:09.761017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:60935)
0926 05:47:09.761184 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:09.761226 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260926 05:47:09.762737 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:09.762500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:51403)
0926 05:47:09.762654 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:09.762694 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260926 05:47:10.004477 19667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:10.004585 19667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:10.004820 19667 leader_election.cc:288] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167), 7194a79e54c2417aa0a001e40337d8df (127.11.247.60:36611)
I20260926 05:47:10.005204 19138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e906df8f6849c293efc589a8d96490" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7194a79e54c2417aa0a001e40337d8df" is_pre_election: true
I20260926 05:47:10.005292 19138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93e906df8f6849c293efc589a8d96490 in term 2.
I20260926 05:47:10.005280 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.005069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:55204)
0926 05:47:10.005187 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.005237 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260926 05:47:10.005450 18927 leader_election.cc:302] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7194a79e54c2417aa0a001e40337d8df, 93e906df8f6849c293efc589a8d96490; no voters:
I20260926 05:47:10.005585 19667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 05:47:10.005633 19667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:10.005656 19667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:47:10.005908 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.005792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:55208)
0926 05:47:10.005847 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.005875 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:47:10.005982 18928 leader_election.cc:334] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:10.006498 19667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
I20260926 05:47:10.006670 19667 leader_election.cc:288] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 3 election: Requested vote from peers a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167), 7194a79e54c2417aa0a001e40337d8df (127.11.247.60:36611)
I20260926 05:47:10.006853 19138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93e906df8f6849c293efc589a8d96490" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7194a79e54c2417aa0a001e40337d8df"
I20260926 05:47:10.006980 19138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:47:10.007620 19138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e906df8f6849c293efc589a8d96490 in term 3.
I20260926 05:47:10.007784 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.006776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:55218)
0926 05:47:10.007696 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.007755 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 05:47:10.007802 18927 leader_election.cc:302] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7194a79e54c2417aa0a001e40337d8df, 93e906df8f6849c293efc589a8d96490; no voters:
I20260926 05:47:10.007896 19667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:47:10.008008 19667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [term 3 LEADER]: Becoming Leader. State: Replica: 93e906df8f6849c293efc589a8d96490, State: Running, Role: LEADER
I20260926 05:47:10.008158 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.008030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:55222)
0926 05:47:10.008082 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.008112 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:47:10.008119 19667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } }
W20260926 05:47:10.008239 18928 leader_election.cc:334] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
I20260926 05:47:10.008519 19670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e906df8f6849c293efc589a8d96490. Latest consensus state: current_term: 3 leader_uuid: "93e906df8f6849c293efc589a8d96490" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:10.008615 19670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:10.008838 19672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:10.009272 19672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:10.009373 19672 catalog_manager.cc:1259] Loaded cluster ID: f491556a2f234ef88e8eb1db84ac071e
I20260926 05:47:10.009428 19672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:10.009626 19672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:10.009729 19672 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Loaded TSK: 0
I20260926 05:47:10.009873 19672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:47:10.041558 18941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:47:10.043464 18941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:10.044615 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.044488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:55240)
0926 05:47:10.044543 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.044579 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:47:10.044651 19138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [term 3 FOLLOWER]: Refusing update from remote peer 93e906df8f6849c293efc589a8d96490: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 05:47:10.044934 19667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 05:47:10.045126 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.045027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.0.0.1:55250)
0926 05:47:10.045061 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.045092 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 05:47:10.045241 18928 consensus_peers.cc:487] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 -> Peer a9cf3850e4ed4be1a1462dbb86082e5b (127.11.247.62:42167): Couldn't send request to peer a9cf3850e4ed4be1a1462dbb86082e5b. Status: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 05:47:10.045581 19674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e906df8f6849c293efc589a8d96490. Latest consensus state: current_term: 3 leader_uuid: "93e906df8f6849c293efc589a8d96490" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:10.045694 19674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:47:10.046233 19667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "93e906df8f6849c293efc589a8d96490" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:10.046334 19667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:10.046289 19674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "93e906df8f6849c293efc589a8d96490" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a9cf3850e4ed4be1a1462dbb86082e5b" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42167 } } peers { permanent_uuid: "93e906df8f6849c293efc589a8d96490" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 46189 } attrs { promote: false } } peers { permanent_uuid: "7194a79e54c2417aa0a001e40337d8df" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 36611 } attrs { promote: false } } }
I20260926 05:47:10.046489 19674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7194a79e54c2417aa0a001e40337d8df [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:47:10.051796 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.048606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.11.247.1:33433)
0926 05:47:10.048736 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:10.048738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:10.048746 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:10.048771 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:10.048771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:10.048781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:10.048873 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:10.049031 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.049575 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.050893 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.051433 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051516 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.051530 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:10.051684 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 05:47:10.051726 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260926 05:47:10.051796 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.048365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33433 (local address 127.0.0.1:43560)
0926 05:47:10.048515 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.048525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:10.048577 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:10.048879 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:10.048880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:10.048882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:10.048946 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:10.048948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.049670 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.049671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:10.050797 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:10.050799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.051525 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:10.051587 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.051592 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:47:10.051674 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:47:10.051719 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 05:47:10.052139 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.048453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.11.247.3:40379)
0926 05:47:10.048728 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:10.048730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:10.048739 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:10.048769 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:10.048770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:10.048781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:10.048989 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:10.049177 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.049927 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051054 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.051594 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051660 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.051672 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:10.052015 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 05:47:10.052069 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":55,"threads_started":1}
I20260926 05:47:10.052202 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.048573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.11.247.2:38785)
0926 05:47:10.048671 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:10.048673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:10.048875 (+ 202us) server_negotiation.cc:407] Connection header received
0926 05:47:10.048903 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:10.048904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:10.048915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:10.048980 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:10.049144 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.049922 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051269 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.051834 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051890 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.051903 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:10.052078 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:47:10.052125 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
I20260926 05:47:10.051955 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.048390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:40379 (local address 127.0.0.1:34642)
0926 05:47:10.048579 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.048591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:10.048660 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:10.049027 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:10.049027 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:10.049029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:10.049082 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:10.049083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.050021 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.050022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:10.050952 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:10.050954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.051703 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051715 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:10.051776 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.051780 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:10.051828 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:47:10.051875 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":21,"thread_start_us":71,"threads_started":1}
I20260926 05:47:10.052469 19241 tablet_service.cc:1467] Processing CreateTablet for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 (DEFAULT_TABLE table=test-table [id=a67a455b87ca46fc9846c44492b49109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:10.052594 19514 tablet_service.cc:1467] Processing CreateTablet for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 (DEFAULT_TABLE table=test-table [id=a67a455b87ca46fc9846c44492b49109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:10.052626 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:10.052704 19514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:10.052146 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.048458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:38785 (local address 127.0.0.1:34494)
0926 05:47:10.048764 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.048774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:10.048821 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:10.049023 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:10.049023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:10.049025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:10.049066 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:10.049067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.050014 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.050015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:10.051182 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:10.051184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.051934 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.051935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:10.051994 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.051998 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:10.052056 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:47:10.052091 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":37,"thread_start_us":72,"threads_started":1}
I20260926 05:47:10.054111 19686 tablet_bootstrap.cc:492] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b: Bootstrap starting.
I20260926 05:47:10.054394 19378 tablet_service.cc:1467] Processing CreateTablet for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 (DEFAULT_TABLE table=test-table [id=a67a455b87ca46fc9846c44492b49109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:10.054481 19686 tablet_bootstrap.cc:654] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:10.054538 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:10.054898 19685 tablet_bootstrap.cc:492] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Bootstrap starting.
I20260926 05:47:10.055243 19685 tablet_bootstrap.cc:654] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:10.055594 19686 tablet_bootstrap.cc:492] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b: No bootstrap required, opened a new log
I20260926 05:47:10.055648 19686 ts_tablet_manager.cc:1397] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:47:10.055886 19686 raft_consensus.cc:348] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.055980 19686 raft_consensus.cc:374] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:10.056013 19686 raft_consensus.cc:729] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e46366f58cb4b23afecfd2e39986d2b, State: Initialized, Role: FOLLOWER
I20260926 05:47:10.056066 19686 consensus_queue.cc:260] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.056238 19686 ts_tablet_manager.cc:1428] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:10.056471 19685 tablet_bootstrap.cc:492] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: No bootstrap required, opened a new log
I20260926 05:47:10.056517 19685 ts_tablet_manager.cc:1397] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:47:10.056586 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.056438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:38111)
0926 05:47:10.056527 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.056557 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 05:47:10.056623 19685 raft_consensus.cc:348] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.056690 19685 raft_consensus.cc:374] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:10.056721 19685 raft_consensus.cc:729] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1589b65ad8544232b5c76d9cb9d44c64, State: Initialized, Role: FOLLOWER
I20260926 05:47:10.056777 19685 consensus_queue.cc:260] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.056932 19685 ts_tablet_manager.cc:1428] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:47:10.057057 19309 heartbeater.cc:503] Master 127.11.247.61:46189 was elected leader, sending a full tablet report...
I20260926 05:47:10.057286 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.057132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:39107)
0926 05:47:10.057226 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.057256 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 05:47:10.057423 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.057335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:45297)
0926 05:47:10.057370 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.057399 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 05:47:10.057469 19689 tablet_bootstrap.cc:492] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Bootstrap starting.
I20260926 05:47:10.057837 19689 tablet_bootstrap.cc:654] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:10.057976 19582 heartbeater.cc:503] Master 127.11.247.61:46189 was elected leader, sending a full tablet report...
I20260926 05:47:10.058152 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.058062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:40355)
0926 05:47:10.058103 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.058127 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 05:47:10.058787 19689 tablet_bootstrap.cc:492] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: No bootstrap required, opened a new log
I20260926 05:47:10.058831 19689 ts_tablet_manager.cc:1397] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:10.058955 19689 raft_consensus.cc:348] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.059007 19689 raft_consensus.cc:374] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:10.059020 19689 raft_consensus.cc:729] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e03719a5bb43f8ade3cfb83377062d, State: Initialized, Role: FOLLOWER
I20260926 05:47:10.059079 19689 consensus_queue.cc:260] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.059283 19689 ts_tablet_manager.cc:1428] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:47:10.059556 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.059442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:39569)
0926 05:47:10.059481 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.059515 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 05:47:10.059720 19446 heartbeater.cc:503] Master 127.11.247.61:46189 was elected leader, sending a full tablet report...
I20260926 05:47:10.059942 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.059801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:58101)
0926 05:47:10.059893 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.059918 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 05:47:10.266163 19691 raft_consensus.cc:482] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:10.266283 19691 raft_consensus.cc:504] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.266459 19691 leader_election.cc:288] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433), 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:10.266722 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaa023a257e4438b9e6b8c6a3ff2b33" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e46366f58cb4b23afecfd2e39986d2b" is_pre_election: true
I20260926 05:47:10.266824 19261 raft_consensus.cc:2454] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 0.
I20260926 05:47:10.266846 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaa023a257e4438b9e6b8c6a3ff2b33" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e03719a5bb43f8ade3cfb83377062d" is_pre_election: true
I20260926 05:47:10.266978 19398 raft_consensus.cc:2454] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 0.
I20260926 05:47:10.267010 19467 leader_election.cc:302] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1589b65ad8544232b5c76d9cb9d44c64, 8e46366f58cb4b23afecfd2e39986d2b; no voters:
I20260926 05:47:10.267153 19691 raft_consensus.cc:2781] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:10.267221 19691 raft_consensus.cc:482] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:10.267237 19691 raft_consensus.cc:3037] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:10.267944 19691 raft_consensus.cc:504] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.268083 19691 leader_election.cc:288] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 election: Requested vote from peers 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433), 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785)
I20260926 05:47:10.268277 19398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaa023a257e4438b9e6b8c6a3ff2b33" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e03719a5bb43f8ade3cfb83377062d"
I20260926 05:47:10.268270 19261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eaa023a257e4438b9e6b8c6a3ff2b33" candidate_uuid: "1589b65ad8544232b5c76d9cb9d44c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e46366f58cb4b23afecfd2e39986d2b"
I20260926 05:47:10.268348 19261 raft_consensus.cc:3037] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:10.268348 19398 raft_consensus.cc:3037] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:10.269080 19398 raft_consensus.cc:2454] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 1.
I20260926 05:47:10.269076 19261 raft_consensus.cc:2454] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1589b65ad8544232b5c76d9cb9d44c64 in term 1.
I20260926 05:47:10.269258 19467 leader_election.cc:302] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1589b65ad8544232b5c76d9cb9d44c64, 8e46366f58cb4b23afecfd2e39986d2b; no voters:
I20260926 05:47:10.269371 19691 raft_consensus.cc:2781] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:10.269524 19691 raft_consensus.cc:686] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 LEADER]: Becoming Leader. State: Replica: 1589b65ad8544232b5c76d9cb9d44c64, State: Running, Role: LEADER
I20260926 05:47:10.269572 19691 consensus_queue.cc:237] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } }
I20260926 05:47:10.269901 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.269764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:59307)
0926 05:47:10.269810 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.269842 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:47:10.270231 18942 catalog_manager.cc:5495] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "1589b65ad8544232b5c76d9cb9d44c64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1589b65ad8544232b5c76d9cb9d44c64" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 40379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 } health_report { overall_health: UNKNOWN } } }
I20260926 05:47:10.279418 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.277048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.11.247.3:40379)
0926 05:47:10.277125 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:10.277127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:10.277137 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:10.277169 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:10.277170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:10.277179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:10.277242 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:10.277378 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.277873 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.278683 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:10.278883 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:10.278972 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:10.279123 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:10.279163 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:10.279225 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:10.279281 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 05:47:10.279323 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:47:10.280414 19261 raft_consensus.cc:1261] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 8e46366f58cb4b23afecfd2e39986d2b [term 1 FOLLOWER]: Refusing update from remote peer 1589b65ad8544232b5c76d9cb9d44c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:10.280526 19398 raft_consensus.cc:1261] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Refusing update from remote peer 1589b65ad8544232b5c76d9cb9d44c64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:10.280726 19691 consensus_queue.cc:1035] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e46366f58cb4b23afecfd2e39986d2b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:10.280885 19694 consensus_queue.cc:1035] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18e03719a5bb43f8ade3cfb83377062d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 38785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:10.281726 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.281603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:44331)
0926 05:47:10.281649 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.281686 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 05:47:10.281834 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.281712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:55655)
0926 05:47:10.281768 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.281799 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 05:47:10.282641 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.282503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:36291)
0926 05:47:10.282549 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.282609 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 05:47:10.283185 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.282884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.2:44375)
0926 05:47:10.282924 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.282945 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:47:10.282437 19698 mvcc.cc:205] Tried to move back new op lower bound from 7333485077626331136 to 7333485077584674816. Current Snapshot: MvccSnapshot[applied={T|T < 7333485077626331136}]
I20260926 05:47:10.283613 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.283517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.3:45981)
0926 05:47:10.283553 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.283587 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 05:47:10.284058 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.283859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42167 (local address 127.11.247.1:47533)
0926 05:47:10.283998 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.284025 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 05:47:10.285266 18942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:10.285351 18942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:10.286617 18942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93e906df8f6849c293efc589a8d96490: Sending DeleteTablet for 3 replicas of tablet 8eaa023a257e4438b9e6b8c6a3ff2b33
I20260926 05:47:10.286952 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:10 UTC) from {username='oryx', principal='oryx/127.11.247.61@KRBTEST.COM'} at 127.0.0.1:34494
I20260926 05:47:10.287093 19241 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:10 UTC) from {username='oryx', principal='oryx/127.11.247.61@KRBTEST.COM'} at 127.0.0.1:43560
I20260926 05:47:10.287176 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 19089
I20260926 05:47:10.287858 19514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:10 UTC) from {username='oryx', principal='oryx/127.11.247.61@KRBTEST.COM'} at 127.0.0.1:34642
I20260926 05:47:10.288038 19708 tablet_replica.cc:330] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: stopping tablet replica
I20260926 05:47:10.288050 19706 tablet_replica.cc:330] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: stopping tablet replica
I20260926 05:47:10.288106 19706 raft_consensus.cc:2229] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:10.288123 19708 raft_consensus.cc:2229] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:10.288172 19706 raft_consensus.cc:2258] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:10.288234 19708 raft_consensus.cc:2258] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:10.288585 19706 ts_tablet_manager.cc:1905] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:10.289726 19708 ts_tablet_manager.cc:1905] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:10.290344 19706 ts_tablet_manager.cc:1918] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:10.290431 19706 log.cc:1196] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/8eaa023a257e4438b9e6b8c6a3ff2b33
I20260926 05:47:10.290723 19706 ts_tablet_manager.cc:1939] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 18e03719a5bb43f8ade3cfb83377062d: Deleting consensus metadata
I20260926 05:47:10.291049 18926 catalog_manager.cc:4841] TS 18e03719a5bb43f8ade3cfb83377062d (127.11.247.2:38785): tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 (table test-table [id=a67a455b87ca46fc9846c44492b49109]) successfully deleted
I20260926 05:47:10.291739 19708 ts_tablet_manager.cc:1918] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:10.291814 19708 log.cc:1196] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/8eaa023a257e4438b9e6b8c6a3ff2b33
I20260926 05:47:10.292043 19708 ts_tablet_manager.cc:1939] T 8eaa023a257e4438b9e6b8c6a3ff2b33 P 1589b65ad8544232b5c76d9cb9d44c64: Deleting consensus metadata
I20260926 05:47:10.292398 18928 catalog_manager.cc:4841] TS 1589b65ad8544232b5c76d9cb9d44c64 (127.11.247.3:40379): tablet 8eaa023a257e4438b9e6b8c6a3ff2b33 (table test-table [id=a67a455b87ca46fc9846c44492b49109]) successfully deleted
I20260926 05:47:10.293947 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:10.293815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33433 (local address 127.0.0.1:43912)
0926 05:47:10.293876 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:10.293908 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:33433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 05:47:10.293937 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 19315
W20260926 05:47:10.295640 18926 catalog_manager.cc:4584] TS 8e46366f58cb4b23afecfd2e39986d2b (127.11.247.1:33433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8eaa023a257e4438b9e6b8c6a3ff2b33: Network error: Client connection negotiation failed: client connection to 127.11.247.1:33433: connect: Connection refused (error 111)
I20260926 05:47:10.299126 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 19451
I20260926 05:47:10.304917 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18808
I20260926 05:47:10.310177 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 18996
2026-09-26T05:47:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6794 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:10 dist-test-slave-21p4 krb5kdc[19711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:10 dist-test-slave-21p4 krb5kdc[19711](info): set up 2 sockets
Sep 26 05:47:10 dist-test-slave-21p4 krb5kdc[19711](info): commencing operation
krb5kdc: starting...
W20260926 05:47:11.383184 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:11 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab.
Entry for principal oryx/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab.
I20260926 05:47:11.510865 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:45197
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab
--principal=oryx/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:11.593571 19724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:11.593848 19724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:11.593909 19724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:11.595151 19724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:11.595233 19724 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:47:11.595255 19724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:11.595278 19724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:11.595304 19724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:11.595338 19724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:11.597081 19724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:45197
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.62.keytab
--principal=oryx/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:11.597401 19724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:11.597678 19724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:11.602766 19729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:11.602782 19732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:11.602782 19730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:11.603192 19724 server_base.cc:1034] running on GCE node
I20260926 05:47:11.603454 19724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:11.603768 19724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:11.604951 19724 hybrid_clock.cc:648] HybridClock initialized: now 1790401631604938 us; error 46 us; skew 500 ppm
Sep 26 05:47:11 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:11.607726 19724 init.cc:377] Logged in from keytab as oryx/127.11.247.62@KRBTEST.COM (short username oryx)
I20260926 05:47:11.608214 19724 webserver.cc:460] Webserver started at http://127.11.247.62:38289/ using document root <none> and password file <none>
I20260926 05:47:11.608476 19724 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:11.608558 19724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:11.608717 19724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:11.609839 19724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "cd066e1173da44bb8a970ef05ad3c863"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "e615565d61e11e8fa23d2dddf7b4ae67"
server_key_iv: "be078abf61f77b888eedaceb9f353f0d"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.610226 19724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "cd066e1173da44bb8a970ef05ad3c863"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "e615565d61e11e8fa23d2dddf7b4ae67"
server_key_iv: "be078abf61f77b888eedaceb9f353f0d"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.611977 19724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:11.612916 19739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:11.613158 19724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:11.613287 19724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "cd066e1173da44bb8a970ef05ad3c863"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "e615565d61e11e8fa23d2dddf7b4ae67"
server_key_iv: "be078abf61f77b888eedaceb9f353f0d"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.613374 19724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:11.625090 19724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:11.628230 19724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:11.628437 19724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:11.633579 19724 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:45197
I20260926 05:47:11.633764 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:45197 every 8 connection(s)
I20260926 05:47:11.634060 19724 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:11.634982 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:11.636097 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 19724
I20260926 05:47:11.636207 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:11.636401 12252 external_mini_cluster.cc:1428] Setting key cc3f7c774bcb34a5881707f7dd9e844d
I20260926 05:47:11.637655 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Bootstrap starting.
I20260926 05:47:11.638253 19802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:11.638697 19802 log.cc:824] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:11.639487 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: No bootstrap required, opened a new log
I20260926 05:47:11.641325 19802 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd066e1173da44bb8a970ef05ad3c863" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45197 } }
I20260926 05:47:11.641498 19802 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:11.641541 19802 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd066e1173da44bb8a970ef05ad3c863, State: Initialized, Role: FOLLOWER
I20260926 05:47:11.641661 19802 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd066e1173da44bb8a970ef05ad3c863" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45197 } }
I20260926 05:47:11.641731 19802 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:11.641760 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:11.641814 19802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:11.642462 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd066e1173da44bb8a970ef05ad3c863" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45197 } }
I20260926 05:47:11.642560 19802 leader_election.cc:302] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd066e1173da44bb8a970ef05ad3c863; no voters:
I20260926 05:47:11.642796 19802 leader_election.cc:288] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:11.642832 19807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:11.642948 19807 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [term 1 LEADER]: Becoming Leader. State: Replica: cd066e1173da44bb8a970ef05ad3c863, State: Running, Role: LEADER
I20260926 05:47:11.643039 19807 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd066e1173da44bb8a970ef05ad3c863" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45197 } }
I20260926 05:47:11.643079 19802 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:11.643347 19807 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd066e1173da44bb8a970ef05ad3c863. Latest consensus state: current_term: 1 leader_uuid: "cd066e1173da44bb8a970ef05ad3c863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd066e1173da44bb8a970ef05ad3c863" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45197 } } }
I20260926 05:47:11.643427 19807 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:11.643359 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd066e1173da44bb8a970ef05ad3c863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd066e1173da44bb8a970ef05ad3c863" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45197 } } }
I20260926 05:47:11.643607 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:11.643630 19811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:11.644172 19811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:47:11 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:11.647078 19811 catalog_manager.cc:1347] Generated new cluster ID: bbb04f6916aa4bfb8c5fd01bb1219c16
I20260926 05:47:11.647140 19811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:11.650797 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:11.636990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.11.247.62:45197)
0926 05:47:11.638705 (+ 1715us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:11.638715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:11.638749 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:47:11.639354 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:11.639363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:11.639428 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:11.639593 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:11.640300 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.642087 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.643202 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.643887 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.643949 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:11.647297 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:11.647302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:11.647302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:11.647321 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:11.649108 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:11.649501 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:11.649505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:11.649505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:11.649602 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:11.649831 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:11.649832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:11.649833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:11.650108 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:11.650296 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:11.650396 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 05:47:11.650550 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":77,"threads_started":1}
I20260926 05:47:11.677541 19811 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:11.678357 19811 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:11.686053 19811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Generated new TSK 0
I20260926 05:47:11.686314 19811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab.
Entry for principal oryx/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab.
I20260926 05:47:11.719329 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45197
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:11.807919 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:11.808221 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:11.808270 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:11.809576 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:11.809666 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:11.809711 19828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:47:11.809753 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:11.811517 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:11.811841 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:11.812103 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:11.817921 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:11.817950 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:11.817955 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:11.817971 19828 server_base.cc:1034] running on GCE node
I20260926 05:47:11.818831 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:11.819156 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:11.820370 19828 hybrid_clock.cc:648] HybridClock initialized: now 1790401631820305 us; error 94 us; skew 500 ppm
Sep 26 05:47:11 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:11.823328 19828 init.cc:377] Logged in from keytab as oryx/127.11.247.1@KRBTEST.COM (short username oryx)
I20260926 05:47:11.823850 19828 webserver.cc:460] Webserver started at http://127.11.247.1:44983/ using document root <none> and password file <none>
I20260926 05:47:11.824127 19828 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:11.824177 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:11.824314 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:11.825516 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "aae1612a24c84953ab304b995f26b151"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "5f99a184e4dda8e238671b24b4daf973"
server_key_iv: "4fb3e226e1d0aa93bb9ea2c01b3865ae"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.825891 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "aae1612a24c84953ab304b995f26b151"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "5f99a184e4dda8e238671b24b4daf973"
server_key_iv: "4fb3e226e1d0aa93bb9ea2c01b3865ae"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.827463 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:47:11.828338 19843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:11.828577 19828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:11.828706 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "aae1612a24c84953ab304b995f26b151"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "5f99a184e4dda8e238671b24b4daf973"
server_key_iv: "4fb3e226e1d0aa93bb9ea2c01b3865ae"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.828796 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:11.842682 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:11.845429 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:11.845646 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:11.846246 19828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:11.846313 19828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:11.846362 19828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:11.846421 19828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:11.855064 19828 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:36505
I20260926 05:47:11.855151 19958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:36505 every 8 connection(s)
Sep 26 05:47:11 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:11.855535 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:11.859090 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:11.846925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:49075 (local address 127.11.247.62:45197)
0926 05:47:11.846974 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:11.846976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:11.847358 (+ 382us) server_negotiation.cc:407] Connection header received
0926 05:47:11.848129 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:11.848130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:11.848184 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:11.848311 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:11.849305 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.850260 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.852225 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.852613 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.852692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:11.856165 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:11.856167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:11.856167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:11.856176 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:11.857588 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:11.858043 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:11.858044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:11.858044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:11.858104 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:11.858348 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:11.858348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:11.858349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:11.858462 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:11.858560 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:11.858931 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 05:47:11.859017 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:47:11.859227 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:11.846931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.11.247.1:49075)
0926 05:47:11.847135 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:11.847208 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:11.847964 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:11.848426 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:11.848431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:11.848847 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:11.849168 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:11.849175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.850408 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.850409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:11.852095 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:11.852100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.852774 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.852775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:11.852956 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:11.853834 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:11.853847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:11.856038 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:11.857686 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:11.857695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:11.857698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:11.857960 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:11.858173 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:11.858174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:11.858175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:11.858288 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:11.858556 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:11.858565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:11.858776 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 05:47:11.858871 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:47:11.859008 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 05:47:11.859825 19959 heartbeater.cc:347] Connected to a master server at 127.11.247.62:45197
I20260926 05:47:11.859980 19959 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:11.860256 19959 heartbeater.cc:511] Master 127.11.247.62:45197 requested a full tablet report, sending...
I20260926 05:47:11.860898 19755 ts_manager.cc:194] Registered new tserver with Master: aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505)
I20260926 05:47:11.861873 19755 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:49075
I20260926 05:47:11.865825 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 19828
I20260926 05:47:11.865975 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:11.866243 12252 external_mini_cluster.cc:1428] Setting key 75b38baecef782c8124d310e9ef0d359
I20260926 05:47:11.867614 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:11.861083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:34851 (local address 127.11.247.62:45197)
0926 05:47:11.861129 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:11.861131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:11.861221 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:47:11.861302 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:11.861303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:11.861311 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:11.861361 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:11.861768 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.862438 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.863268 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.863475 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.863567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:11.864556 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:11.864558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:11.864558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:11.864566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:11.866219 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:11.866731 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:11.866732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:11.866733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:11.866799 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:11.867006 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:11.867007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:11.867007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:11.867181 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:11.867263 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:11.867465 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 05:47:11.867532 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:47:11.867759 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:11.860994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.11.247.1:34851)
0926 05:47:11.861162 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:11.861174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:11.861239 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:11.861367 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:11.861368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:11.861597 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:11.861670 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:11.861672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.862543 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.862545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:11.863162 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:11.863164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:11.863573 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:11.863574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:11.863648 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:11.864121 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:11.864125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:11.864464 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:11.866327 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:11.866328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:11.866329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:11.866634 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:11.866870 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:11.866871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:11.866872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:11.866935 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:11.867268 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:11.867269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:11.867344 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 05:47:11.867448 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:47:11.867581 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab.
Entry for principal oryx/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab.
I20260926 05:47:11.895153 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45197
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:11.980890 19968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:11.981168 19968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:11.981213 19968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:11.982494 19968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:11.982581 19968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:11.982664 19968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:47:11.982705 19968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:11.984351 19968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:11.984679 19968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:11.984934 19968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:47:11.990846 19968 server_base.cc:1034] running on GCE node
W20260926 05:47:11.990863 19973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:11.990856 19976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:11.991181 19974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:11.991482 19968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:11.991768 19968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:11.992976 19968 hybrid_clock.cc:648] HybridClock initialized: now 1790401631992968 us; error 40 us; skew 500 ppm
Sep 26 05:47:11 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:11.995860 19968 init.cc:377] Logged in from keytab as oryx/127.11.247.2@KRBTEST.COM (short username oryx)
I20260926 05:47:11.996438 19968 webserver.cc:460] Webserver started at http://127.11.247.2:42791/ using document root <none> and password file <none>
I20260926 05:47:11.996706 19968 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:11.996779 19968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:11.996938 19968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:11.998049 19968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "d88c693f46144b878b40a533abff39b9"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "75f293b127d3ca01d29fb02081c21008"
server_key_iv: "2b590ffd7dafb65ea426dd9a23997354"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.998451 19968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "d88c693f46144b878b40a533abff39b9"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "75f293b127d3ca01d29fb02081c21008"
server_key_iv: "2b590ffd7dafb65ea426dd9a23997354"
server_key_version: "encryptionkey@0"
I20260926 05:47:11.999990 19968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:12.000825 19983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:12.001053 19968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:12.001175 19968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "d88c693f46144b878b40a533abff39b9"
format_stamp: "Formatted at 2026-09-26 05:47:11 on dist-test-slave-21p4"
server_key: "75f293b127d3ca01d29fb02081c21008"
server_key_iv: "2b590ffd7dafb65ea426dd9a23997354"
server_key_version: "encryptionkey@0"
I20260926 05:47:12.001262 19968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:12.026933 19968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:12.029310 19968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:12.029506 19968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:12.030257 19968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:12.030310 19968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:12.030357 19968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:12.030371 19968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:12.038420 19968 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:34113
I20260926 05:47:12.038512 20098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:34113 every 8 connection(s)
I20260926 05:47:12.038872 19968 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:47:12.041029 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 19968
I20260926 05:47:12.041173 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:12.041407 12252 external_mini_cluster.cc:1428] Setting key 5fd8b99b0df9e02bf8b59a0aabe83a22
I20260926 05:47:12.042070 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.030859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:44861 (local address 127.11.247.62:45197)
0926 05:47:12.030936 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.030939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.031220 (+ 281us) server_negotiation.cc:407] Connection header received
0926 05:47:12.031868 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.031869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.031891 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.031972 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.033011 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.033631 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.034999 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.035267 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.035361 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.038397 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:12.038399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:12.038400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:12.038409 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:12.040215 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.040695 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.040697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.040697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.040764 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.041256 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.041257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.041258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.041401 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:12.041487 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.041873 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.041968 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:47:12.042339 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.030861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.11.247.2:44861)
0926 05:47:12.031063 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.031112 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.031705 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.032080 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.032084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.032486 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:12.032859 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.032869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.033777 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.033779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.034876 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.034879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.035411 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.035412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.035560 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.036279 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:12.036289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:12.038271 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:12.040310 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.040316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.040318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.040568 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.040939 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.040939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.040940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.041059 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.041476 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:12.041483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:12.041703 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.041845 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.042040 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":119,"threads_started":1}
I20260926 05:47:12.042923 20099 heartbeater.cc:347] Connected to a master server at 127.11.247.62:45197
I20260926 05:47:12.043011 20099 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:12.043241 20099 heartbeater.cc:511] Master 127.11.247.62:45197 requested a full tablet report, sending...
I20260926 05:47:12.043687 19756 ts_manager.cc:194] Registered new tserver with Master: d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:12.044212 19756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:44861
WARNING: no policy specified for oryx/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:47:12.052309 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.044243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:52983 (local address 127.11.247.62:45197)
0926 05:47:12.044290 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.044292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.045333 (+ 1041us) server_negotiation.cc:407] Connection header received
0926 05:47:12.045363 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.045364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.045374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.045430 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.045852 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.046349 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.047538 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.047817 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.047887 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.049071 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:12.049072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:12.049072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:12.049079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:12.050868 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.051287 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.051288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.051289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.051357 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.051598 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.051599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.051600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.051806 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:12.051895 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.052102 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.052196 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:47:12.052322 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.044066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.11.247.2:52983)
0926 05:47:12.045240 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.045252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.045322 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.045436 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.045436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.045673 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:12.045757 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.045759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.046463 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.046465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.047426 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.047430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.047907 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.047908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.047988 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.048490 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:12.048496 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:12.048972 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:12.050984 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.050986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.050986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.051202 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.051433 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.051434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.051435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.051503 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.051900 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:12.051901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:12.051970 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.052072 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.052202 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab.
Entry for principal oryx/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab.
I20260926 05:47:12.071055 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45197
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:12.159096 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:12.159411 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:12.159453 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:12.160832 20108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:12.160885 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:12.160928 20108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:47:12.160948 20108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:12.162637 20108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.247.62:45197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:12.162875 20108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:12.163071 20108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:12.168358 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:12.168354 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:12.168358 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:12.168848 20108 server_base.cc:1034] running on GCE node
I20260926 05:47:12.169096 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:12.169376 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:12.170555 20108 hybrid_clock.cc:648] HybridClock initialized: now 1790401632170550 us; error 41 us; skew 500 ppm
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:12.173216 20108 init.cc:377] Logged in from keytab as oryx/127.11.247.3@KRBTEST.COM (short username oryx)
I20260926 05:47:12.173731 20108 webserver.cc:460] Webserver started at http://127.11.247.3:43795/ using document root <none> and password file <none>
I20260926 05:47:12.174005 20108 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:12.174054 20108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:12.174226 20108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:12.175401 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "abc3bc947aeb4d8da7643aa62b0d8d22"
format_stamp: "Formatted at 2026-09-26 05:47:12 on dist-test-slave-21p4"
server_key: "789cc6afd3874aeade3681a719d730c3"
server_key_iv: "d0cf69b4286dccd45821c143409f3129"
server_key_version: "encryptionkey@0"
I20260926 05:47:12.175774 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "abc3bc947aeb4d8da7643aa62b0d8d22"
format_stamp: "Formatted at 2026-09-26 05:47:12 on dist-test-slave-21p4"
server_key: "789cc6afd3874aeade3681a719d730c3"
server_key_iv: "d0cf69b4286dccd45821c143409f3129"
server_key_version: "encryptionkey@0"
I20260926 05:47:12.177248 20108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:47:12.178213 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:12.178560 20108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:12.178761 20108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "abc3bc947aeb4d8da7643aa62b0d8d22"
format_stamp: "Formatted at 2026-09-26 05:47:12 on dist-test-slave-21p4"
server_key: "789cc6afd3874aeade3681a719d730c3"
server_key_iv: "d0cf69b4286dccd45821c143409f3129"
server_key_version: "encryptionkey@0"
I20260926 05:47:12.178850 20108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:12.195122 20108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:12.197678 20108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:12.197880 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:12.198565 20108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:12.198679 20108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:12.198717 20108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:12.198765 20108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:12.206107 20108 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:38769
I20260926 05:47:12.206207 20238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:38769 every 8 connection(s)
I20260926 05:47:12.206547 20108 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:12.206955 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20108
I20260926 05:47:12.207060 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:12.207276 12252 external_mini_cluster.cc:1428] Setting key 52b6ec85f9ad60c0f41cab8d33fd1ae9
I20260926 05:47:12.209673 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.199005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:49729 (local address 127.11.247.62:45197)
0926 05:47:12.199072 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.199075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.199389 (+ 314us) server_negotiation.cc:407] Connection header received
0926 05:47:12.199996 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.199997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.200009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.200074 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.200836 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.201357 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.202454 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.202759 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.202846 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.205911 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:12.205913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:12.205913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:12.205931 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:12.208100 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.208582 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.208583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.208583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.208649 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.208910 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.208911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.208912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.209099 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:12.209173 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.209480 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.209581 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:47:12.209872 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.199011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.11.247.3:49729)
0926 05:47:12.199245 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.199288 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.199883 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.200157 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.200160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.200465 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:12.200733 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.200739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.201460 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.201461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.202349 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.202352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.202856 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.202858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.203036 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.203580 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:12.203588 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:12.205793 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:12.208184 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.208192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.208194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.208477 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.208725 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.208726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.208727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.208842 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.209175 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:12.209182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:12.209335 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.209445 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.209625 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260926 05:47:12.210383 20239 heartbeater.cc:347] Connected to a master server at 127.11.247.62:45197
I20260926 05:47:12.210472 20239 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:12.210711 20239 heartbeater.cc:511] Master 127.11.247.62:45197 requested a full tablet report, sending...
I20260926 05:47:12.211143 19755 ts_manager.cc:194] Registered new tserver with Master: abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769)
I20260926 05:47:12.211687 19755 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:49729
I20260926 05:47:12.217932 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.211672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:54479 (local address 127.11.247.62:45197)
0926 05:47:12.211719 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.211721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.211798 (+ 77us) server_negotiation.cc:407] Connection header received
0926 05:47:12.211901 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.211901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.211912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.211971 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.212299 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.212871 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.213665 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.213864 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.213930 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.214994 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:12.214995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:12.214995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:12.215001 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:12.216723 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.217108 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.217109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.217109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.217170 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.217376 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.217376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.217377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.217504 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:12.217557 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.217801 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.217867 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:47:12.217944 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.211609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.11.247.3:54479)
0926 05:47:12.211731 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.211743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.211827 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.211983 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.211984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.212159 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:12.212219 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.212221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.212979 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.212980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.213533 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.213535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.213943 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.213944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.214001 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.214488 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:12.214493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:12.214908 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:12.216811 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.216812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.216812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.216985 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.217239 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.217240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.217240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.217295 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.217639 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:12.217641 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:12.217699 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.217766 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.217862 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260926 05:47:12.219424 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:47:12.242496 19805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:12.236371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.11.247.62:45197)
0926 05:47:12.236459 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.236461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.236483 (+ 22us) server_negotiation.cc:407] Connection header received
0926 05:47:12.236513 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.236514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.236523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.236585 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.237281 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.237808 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.238739 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.238928 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.238983 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.242405 (+ 3422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40314: BlockingRecv error: recv got EOF from 127.0.0.1:40314 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401632, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:12.251261 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.243487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.11.247.62:45197)
0926 05:47:12.243553 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.243556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.243610 (+ 54us) server_negotiation.cc:407] Connection header received
0926 05:47:12.243739 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.243740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.243751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.243828 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.244401 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.245008 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.245718 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.245912 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.245991 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.248429 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:12.248431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:12.248431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:12.248439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:12.249935 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.250327 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.250328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.250329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.250392 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.250740 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.250741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.250741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.250883 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:12.250972 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.251119 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.251189 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:47:12.253185 19755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:47:12.254968 19755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:12.262847 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.258011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:38769 (local address 127.0.0.1:45134)
0926 05:47:12.258159 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.258208 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.258283 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.258721 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.258725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.258728 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.258827 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.258830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.260278 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.260278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.261461 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.261464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.262371 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.262371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.262448 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.262461 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.262557 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.262647 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260926 05:47:12.262852 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.258309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.11.247.3:38769)
0926 05:47:12.258473 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.258478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.258489 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:12.258534 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.258536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.258558 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.258918 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.258982 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.260179 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.261538 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.262278 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.262359 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.262446 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.262616 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.262657 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260926 05:47:12.263510 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.258114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36505 (local address 127.0.0.1:49862)
0926 05:47:12.258676 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.258685 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.258733 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.259094 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.259095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.259097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.259147 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.259148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.260338 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.260339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.261376 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.261380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.263269 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.263270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.263338 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.263343 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.263400 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.263444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":69,"thread_start_us":117,"threads_started":1}
I20260926 05:47:12.263782 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.258374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.11.247.2:34113)
0926 05:47:12.258541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.258545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.258659 (+ 114us) server_negotiation.cc:407] Connection header received
0926 05:47:12.258762 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.258764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.258781 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.258854 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.259093 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.259932 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.262487 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.263197 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.263262 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.263335 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.263592 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.263685 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260926 05:47:12.263698 20173 tablet_service.cc:1467] Processing CreateTablet for tablet 6987ab2601dc4e51b8365104daa8b7f4 (DEFAULT_TABLE table=test-table [id=fd997e001e8c417cb922e7b522158b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:12.263895 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.258392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.11.247.1:36505)
0926 05:47:12.258619 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.258624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.258793 (+ 169us) server_negotiation.cc:407] Connection header received
0926 05:47:12.258839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.258841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.258953 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.259031 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.259226 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.260270 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.261470 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.262278 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.262359 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.262445 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.263752 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.263801 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260926 05:47:12.264068 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.258113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:34113 (local address 127.0.0.1:58100)
0926 05:47:12.258551 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.258563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.258648 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.258949 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.258950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.258952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.259013 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.259014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.260016 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.260017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.261461 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.261464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.263372 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.263373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.263442 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.263446 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.263495 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.263544 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":107,"threads_started":1}
I20260926 05:47:12.264150 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6987ab2601dc4e51b8365104daa8b7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.264501 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 6987ab2601dc4e51b8365104daa8b7f4 (DEFAULT_TABLE table=test-table [id=fd997e001e8c417cb922e7b522158b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:12.264849 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6987ab2601dc4e51b8365104daa8b7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.267230 20033 tablet_service.cc:1467] Processing CreateTablet for tablet 6987ab2601dc4e51b8365104daa8b7f4 (DEFAULT_TABLE table=test-table [id=fd997e001e8c417cb922e7b522158b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:12.267685 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6987ab2601dc4e51b8365104daa8b7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.268287 20267 tablet_bootstrap.cc:492] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Bootstrap starting.
I20260926 05:47:12.268999 20268 tablet_bootstrap.cc:492] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Bootstrap starting.
I20260926 05:47:12.269145 20267 tablet_bootstrap.cc:654] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.269533 20268 tablet_bootstrap.cc:654] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.269603 20267 log.cc:824] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:12.269892 20268 log.cc:824] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:12.270481 20267 tablet_bootstrap.cc:492] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: No bootstrap required, opened a new log
I20260926 05:47:12.270565 20267 ts_tablet_manager.cc:1397] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:12.270658 20268 tablet_bootstrap.cc:492] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: No bootstrap required, opened a new log
I20260926 05:47:12.270727 20268 ts_tablet_manager.cc:1397] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:12.272169 20268 raft_consensus.cc:348] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.272339 20268 raft_consensus.cc:374] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.272382 20268 raft_consensus.cc:729] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1612a24c84953ab304b995f26b151, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.272503 20268 consensus_queue.cc:260] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.272794 20268 ts_tablet_manager.cc:1428] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:12.272921 19959 heartbeater.cc:503] Master 127.11.247.62:45197 was elected leader, sending a full tablet report...
I20260926 05:47:12.272817 20271 tablet_bootstrap.cc:492] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Bootstrap starting.
I20260926 05:47:12.273084 20267 raft_consensus.cc:348] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.273295 20267 raft_consensus.cc:374] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.273345 20267 raft_consensus.cc:729] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.273484 20267 consensus_queue.cc:260] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.273785 20267 ts_tablet_manager.cc:1428] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:47:12.273829 20239 heartbeater.cc:503] Master 127.11.247.62:45197 was elected leader, sending a full tablet report...
I20260926 05:47:12.273962 20271 tablet_bootstrap.cc:654] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.274295 20271 log.cc:824] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:12.275277 20271 tablet_bootstrap.cc:492] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: No bootstrap required, opened a new log
I20260926 05:47:12.275382 20271 ts_tablet_manager.cc:1397] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 05:47:12.276980 20271 raft_consensus.cc:348] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.277140 20271 raft_consensus.cc:374] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.277168 20271 raft_consensus.cc:729] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88c693f46144b878b40a533abff39b9, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.277336 20271 consensus_queue.cc:260] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.277593 20271 ts_tablet_manager.cc:1428] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:47:12.277719 20099 heartbeater.cc:503] Master 127.11.247.62:45197 was elected leader, sending a full tablet report...
W20260926 05:47:12.289852 20100 tablet.cc:2369] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:12.314363 20272 raft_consensus.cc:482] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:12.314498 20272 raft_consensus.cc:504] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.314889 20272 leader_election.cc:288] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769), d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:12.319160 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.315144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:38769 (local address 127.11.247.1:58423)
0926 05:47:12.315329 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.315338 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.315417 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.315552 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.315552 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.315554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.315682 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.315684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.316569 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.316570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.318038 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.318043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.318912 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.318914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.318982 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.318986 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.319036 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.319073 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":68,"thread_start_us":112,"threads_started":1}
I20260926 05:47:12.319182 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.315194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:58423 (local address 127.11.247.3:38769)
0926 05:47:12.315246 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.315248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.315403 (+ 155us) server_negotiation.cc:407] Connection header received
0926 05:47:12.315482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.315483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.315491 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.315532 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.315752 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.316483 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.318276 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.318831 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.318893 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.318902 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.319069 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.319110 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:47:12.319746 20193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6987ab2601dc4e51b8365104daa8b7f4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" is_pre_election: true
I20260926 05:47:12.319862 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.315127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:34113 (local address 127.11.247.1:37479)
0926 05:47:12.315198 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.315212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.315269 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.315442 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.315443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.315445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.315551 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.315553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.316506 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.316507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.318038 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.318043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.319649 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.319650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.319719 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.319722 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.319769 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.319806 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":66}
I20260926 05:47:12.319913 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.315218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:37479 (local address 127.11.247.2:34113)
0926 05:47:12.315261 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.315263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.315273 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:12.315346 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.315347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.315356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.315416 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.315630 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.316391 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.318937 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.319489 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.319599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.319608 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.319803 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.319849 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:47:12.319960 20193 raft_consensus.cc:2454] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 0.
I20260926 05:47:12.320225 19844 leader_election.cc:302] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, abc3bc947aeb4d8da7643aa62b0d8d22; no voters:
I20260926 05:47:12.320369 20272 raft_consensus.cc:2781] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:12.320431 20272 raft_consensus.cc:482] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:12.320456 20272 raft_consensus.cc:3037] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.320469 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6987ab2601dc4e51b8365104daa8b7f4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9" is_pre_election: true
I20260926 05:47:12.320608 20053 raft_consensus.cc:2454] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 0.
I20260926 05:47:12.321537 20272 raft_consensus.cc:504] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.321759 20272 leader_election.cc:288] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 election: Requested vote from peers abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769), d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:12.321843 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6987ab2601dc4e51b8365104daa8b7f4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9"
I20260926 05:47:12.321858 20193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6987ab2601dc4e51b8365104daa8b7f4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22"
I20260926 05:47:12.321930 20193 raft_consensus.cc:3037] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.321925 20053 raft_consensus.cc:3037] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.322853 20053 raft_consensus.cc:2454] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 1.
I20260926 05:47:12.322853 20193 raft_consensus.cc:2454] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 1.
I20260926 05:47:12.323038 19844 leader_election.cc:302] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, abc3bc947aeb4d8da7643aa62b0d8d22; no voters:
I20260926 05:47:12.323177 20272 raft_consensus.cc:2781] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:12.323369 20272 raft_consensus.cc:686] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 1 LEADER]: Becoming Leader. State: Replica: aae1612a24c84953ab304b995f26b151, State: Running, Role: LEADER
I20260926 05:47:12.323483 20272 consensus_queue.cc:237] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } }
I20260926 05:47:12.324460 19755 catalog_manager.cc:5495] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae1612a24c84953ab304b995f26b151 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } health_report { overall_health: HEALTHY } } }
I20260926 05:47:12.338529 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.335971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.11.247.1:36505)
0926 05:47:12.336047 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.336049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.336057 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:12.336101 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.336102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.336130 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.336205 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:12.336369 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.336865 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.337646 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.337867 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.337945 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.338154 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.338278 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.338371 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.338413 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.338436 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:47:12.340642 20193 raft_consensus.cc:1261] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Refusing update from remote peer aae1612a24c84953ab304b995f26b151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:12.340642 20053 raft_consensus.cc:1261] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Refusing update from remote peer aae1612a24c84953ab304b995f26b151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:12.341027 20277 consensus_queue.cc:1035] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:12.341156 20272 consensus_queue.cc:1035] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:12.344316 20281 mvcc.cc:205] Tried to move back new op lower bound from 7333485086063570944 to 7333485085997539328. Current Snapshot: MvccSnapshot[applied={T|T < 7333485086063570944}]
I20260926 05:47:12.344450 20282 mvcc.cc:205] Tried to move back new op lower bound from 7333485086063570944 to 7333485085997539328. Current Snapshot: MvccSnapshot[applied={T|T < 7333485086063570944}]
I20260926 05:47:12.347292 19756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:12.347430 19756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:12.348349 19756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Sending DeleteTablet for 3 replicas of tablet 6987ab2601dc4e51b8365104daa8b7f4
I20260926 05:47:12.348781 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6987ab2601dc4e51b8365104daa8b7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:12 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:49862
I20260926 05:47:12.348780 20173 tablet_service.cc:1514] Processing DeleteTablet for tablet 6987ab2601dc4e51b8365104daa8b7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:12 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45134
I20260926 05:47:12.349032 20290 tablet_replica.cc:330] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: stopping tablet replica
I20260926 05:47:12.349009 19756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:47:12.349156 20290 raft_consensus.cc:2229] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 05:47:12.349169 19756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:12.349525 20290 raft_consensus.cc:2258] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:12.349713 20291 tablet_replica.cc:330] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: stopping tablet replica
I20260926 05:47:12.349819 20291 raft_consensus.cc:2229] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:12.349826 20033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6987ab2601dc4e51b8365104daa8b7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:12 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:58100
I20260926 05:47:12.349952 20291 raft_consensus.cc:2258] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:12.350100 20290 ts_tablet_manager.cc:1905] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:12.350258 20292 tablet_replica.cc:330] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: stopping tablet replica
I20260926 05:47:12.350397 20291 ts_tablet_manager.cc:1905] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:12.350409 20292 raft_consensus.cc:2229] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:12.350639 20292 raft_consensus.cc:2258] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:12.350757 20173 tablet_service.cc:1467] Processing CreateTablet for tablet b2c6f8df6d154701ba2079645d80c402 (DEFAULT_TABLE table=test-table [id=c4036be955854dabb5a2b27aac00c671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:12.350879 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c6f8df6d154701ba2079645d80c402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.351851 20267 tablet_bootstrap.cc:492] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Bootstrap starting.
I20260926 05:47:12.352110 20291 ts_tablet_manager.cc:1918] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:12.352180 20291 log.cc:1196] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/6987ab2601dc4e51b8365104daa8b7f4
I20260926 05:47:12.352306 20267 tablet_bootstrap.cc:654] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.352339 20033 tablet_service.cc:1467] Processing CreateTablet for tablet b2c6f8df6d154701ba2079645d80c402 (DEFAULT_TABLE table=test-table [id=c4036be955854dabb5a2b27aac00c671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:12.352468 20291 ts_tablet_manager.cc:1939] T 6987ab2601dc4e51b8365104daa8b7f4 P aae1612a24c84953ab304b995f26b151: Deleting consensus metadata
I20260926 05:47:12.352449 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c6f8df6d154701ba2079645d80c402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.353111 20292 ts_tablet_manager.cc:1905] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:12.353930 20271 tablet_bootstrap.cc:492] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Bootstrap starting.
I20260926 05:47:12.354097 19742 catalog_manager.cc:4841] TS aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505): tablet 6987ab2601dc4e51b8365104daa8b7f4 (table test-table [id=fd997e001e8c417cb922e7b522158b43]) successfully deleted
I20260926 05:47:12.354315 19891 tablet_service.cc:1467] Processing CreateTablet for tablet b2c6f8df6d154701ba2079645d80c402 (DEFAULT_TABLE table=test-table [id=c4036be955854dabb5a2b27aac00c671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:12.354347 20271 tablet_bootstrap.cc:654] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.354416 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c6f8df6d154701ba2079645d80c402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.354682 20292 ts_tablet_manager.cc:1918] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:12.354754 20292 log.cc:1196] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/6987ab2601dc4e51b8365104daa8b7f4
I20260926 05:47:12.355018 20292 ts_tablet_manager.cc:1939] T 6987ab2601dc4e51b8365104daa8b7f4 P d88c693f46144b878b40a533abff39b9: Deleting consensus metadata
I20260926 05:47:12.355304 20271 tablet_bootstrap.cc:492] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: No bootstrap required, opened a new log
I20260926 05:47:12.355355 20271 ts_tablet_manager.cc:1397] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:12.355479 19741 catalog_manager.cc:4841] TS d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113): tablet 6987ab2601dc4e51b8365104daa8b7f4 (table test-table [id=fd997e001e8c417cb922e7b522158b43]) successfully deleted
I20260926 05:47:12.355501 20271 raft_consensus.cc:348] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.355573 20271 raft_consensus.cc:374] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.355594 20271 raft_consensus.cc:729] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88c693f46144b878b40a533abff39b9, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.355650 20271 consensus_queue.cc:260] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.355741 20271 ts_tablet_manager.cc:1428] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:12.355888 20268 tablet_bootstrap.cc:492] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Bootstrap starting.
I20260926 05:47:12.356166 20290 ts_tablet_manager.cc:1918] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:12.356287 20290 log.cc:1196] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/6987ab2601dc4e51b8365104daa8b7f4
I20260926 05:47:12.356302 20268 tablet_bootstrap.cc:654] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.356590 20290 ts_tablet_manager.cc:1939] T 6987ab2601dc4e51b8365104daa8b7f4 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting consensus metadata
I20260926 05:47:12.356964 20268 tablet_bootstrap.cc:492] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: No bootstrap required, opened a new log
I20260926 05:47:12.357015 20268 ts_tablet_manager.cc:1397] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:12.357179 19740 catalog_manager.cc:4841] TS abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769): tablet 6987ab2601dc4e51b8365104daa8b7f4 (table test-table [id=fd997e001e8c417cb922e7b522158b43]) successfully deleted
I20260926 05:47:12.357189 20268 raft_consensus.cc:348] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.357300 20268 raft_consensus.cc:374] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.357336 20268 raft_consensus.cc:729] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1612a24c84953ab304b995f26b151, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.357388 20268 consensus_queue.cc:260] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.357478 20268 ts_tablet_manager.cc:1428] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:47:12.357689 20267 tablet_bootstrap.cc:492] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: No bootstrap required, opened a new log
I20260926 05:47:12.357779 20267 ts_tablet_manager.cc:1397] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260926 05:47:12.357949 20267 raft_consensus.cc:348] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.358032 20267 raft_consensus.cc:374] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.358060 20267 raft_consensus.cc:729] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.358104 20267 consensus_queue.cc:260] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.358207 20267 ts_tablet_manager.cc:1428] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260926 05:47:12.457355 20240 tablet.cc:2369] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:12.531388 20273 raft_consensus.cc:482] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:12.531497 20273 raft_consensus.cc:504] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.531906 20273 leader_election.cc:288] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505), d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:12.535550 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.532160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36505 (local address 127.11.247.3:45649)
0926 05:47:12.532232 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.532244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.532293 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.532476 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.532478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.532480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.532562 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.532564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.533328 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.533329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.534560 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.534563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.535258 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.535258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.535339 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.535343 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.535408 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.535453 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 05:47:12.535537 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.532255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45649 (local address 127.11.247.1:36505)
0926 05:47:12.532312 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.532314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.532322 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:12.532363 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.532364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.532373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.532466 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.532656 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.533212 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.534712 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.535149 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.535260 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.535270 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.535427 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.535469 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:47:12.535871 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c6f8df6d154701ba2079645d80c402" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae1612a24c84953ab304b995f26b151" is_pre_election: true
I20260926 05:47:12.536064 19894 raft_consensus.cc:2454] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 0.
I20260926 05:47:12.536175 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.532255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:43945 (local address 127.11.247.2:34113)
0926 05:47:12.532303 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.532306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.532425 (+ 119us) server_negotiation.cc:407] Connection header received
0926 05:47:12.532552 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.532553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.532564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.532623 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:12.533556 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.534280 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.535330 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.535811 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.535891 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.535901 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.536060 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.536108 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:47:12.536176 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.532160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:34113 (local address 127.11.247.3:43945)
0926 05:47:12.532347 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.532357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.532446 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.533315 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.533316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.533318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:12.533448 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.533451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.534373 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.534374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.535231 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.535233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.535900 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.535901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.535976 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.535979 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.536050 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.536087 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260926 05:47:12.536391 20126 leader_election.cc:302] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, abc3bc947aeb4d8da7643aa62b0d8d22; no voters:
I20260926 05:47:12.536417 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c6f8df6d154701ba2079645d80c402" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9" is_pre_election: true
I20260926 05:47:12.536512 20053 raft_consensus.cc:2454] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 0.
I20260926 05:47:12.536548 20273 raft_consensus.cc:2781] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:12.536582 20273 raft_consensus.cc:482] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:12.536602 20273 raft_consensus.cc:3037] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.537302 20273 raft_consensus.cc:504] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.537457 20273 leader_election.cc:288] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 election: Requested vote from peers aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505), d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:12.537678 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c6f8df6d154701ba2079645d80c402" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9"
I20260926 05:47:12.537678 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c6f8df6d154701ba2079645d80c402" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae1612a24c84953ab304b995f26b151"
I20260926 05:47:12.537757 20053 raft_consensus.cc:3037] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.537757 19894 raft_consensus.cc:3037] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.538492 20053 raft_consensus.cc:2454] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 1.
I20260926 05:47:12.538651 19894 raft_consensus.cc:2454] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 1.
I20260926 05:47:12.538718 20125 leader_election.cc:302] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc3bc947aeb4d8da7643aa62b0d8d22, d88c693f46144b878b40a533abff39b9; no voters:
I20260926 05:47:12.538897 20273 raft_consensus.cc:2781] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:12.539094 20273 raft_consensus.cc:686] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 LEADER]: Becoming Leader. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Running, Role: LEADER
I20260926 05:47:12.539201 20273 consensus_queue.cc:237] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.540015 19756 catalog_manager.cc:5495] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } health_report { overall_health: HEALTHY } } }
Sep 26 05:47:12 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401631, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.62@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:12.583460 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.575509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.11.247.62:45197)
0926 05:47:12.575627 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.575629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.575807 (+ 178us) server_negotiation.cc:407] Connection header received
0926 05:47:12.575858 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.575858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.575870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.575927 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:12.576416 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.576972 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.577835 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.578141 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.578222 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.580808 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:12.580809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:12.580810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:12.580817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:12.582345 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.582808 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.582809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.582809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.582861 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:12.583018 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:12.583018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:12.583019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:12.583132 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:12.583178 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.583320 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.583390 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 05:47:12.583537 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.575421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45197 (local address 127.0.0.1:40336)
0926 05:47:12.575709 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.575721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.575794 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.575944 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.575945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.576271 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:12.576323 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.576325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.577082 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.577083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.577680 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.577682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.578259 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.578260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.578343 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.579126 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:12.579131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:12.580714 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:12.582435 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.582444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.582446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.582747 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.582921 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:12.582922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:12.582922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:12.582980 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:12.583192 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:12.583199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:12.583249 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.583301 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.583447 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":43,"thread_start_us":187,"threads_started":1}
I20260926 05:47:12.585659 19756 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:40336:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:47:12.585942 19756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:12.587699 20173 tablet_service.cc:1467] Processing CreateTablet for tablet 113b15f7f1a7456ea778217327b4acf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52435dfd16274ee9b608e9b04285af15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:47:12.587714 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 113b15f7f1a7456ea778217327b4acf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52435dfd16274ee9b608e9b04285af15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:47:12.587888 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113b15f7f1a7456ea778217327b4acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.587898 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113b15f7f1a7456ea778217327b4acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.589130 20268 tablet_bootstrap.cc:492] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: Bootstrap starting.
I20260926 05:47:12.589272 20033 tablet_service.cc:1467] Processing CreateTablet for tablet 113b15f7f1a7456ea778217327b4acf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52435dfd16274ee9b608e9b04285af15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:47:12.589457 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113b15f7f1a7456ea778217327b4acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:12.589510 20268 tablet_bootstrap.cc:654] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.589808 20267 tablet_bootstrap.cc:492] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: Bootstrap starting.
I20260926 05:47:12.590193 20267 tablet_bootstrap.cc:654] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.590688 20268 tablet_bootstrap.cc:492] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: No bootstrap required, opened a new log
I20260926 05:47:12.590744 20268 ts_tablet_manager.cc:1397] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:47:12.590857 20271 tablet_bootstrap.cc:492] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: Bootstrap starting.
I20260926 05:47:12.590901 20268 raft_consensus.cc:348] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.590981 20268 raft_consensus.cc:374] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.591006 20268 raft_consensus.cc:729] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1612a24c84953ab304b995f26b151, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.591076 20268 consensus_queue.cc:260] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.591177 20268 ts_tablet_manager.cc:1428] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:12.591251 20271 tablet_bootstrap.cc:654] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:12.591218 20277 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.591318 20277 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:47:12.591392 20267 tablet_bootstrap.cc:492] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: No bootstrap required, opened a new log
I20260926 05:47:12.591439 20267 ts_tablet_manager.cc:1397] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:47:12.591751 20267 raft_consensus.cc:348] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.591842 20267 raft_consensus.cc:374] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.591866 20267 raft_consensus.cc:729] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.591918 20267 consensus_queue.cc:260] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.592016 20267 ts_tablet_manager.cc:1428] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:47:12.592124 20271 tablet_bootstrap.cc:492] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: No bootstrap required, opened a new log
I20260926 05:47:12.592082 20273 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.592177 20271 ts_tablet_manager.cc:1397] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:12.592185 20273 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:47:12.592329 20271 raft_consensus.cc:348] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.592401 20271 raft_consensus.cc:374] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:12.592430 20271 raft_consensus.cc:729] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88c693f46144b878b40a533abff39b9, State: Initialized, Role: FOLLOWER
I20260926 05:47:12.592502 20271 consensus_queue.cc:260] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.592607 20271 ts_tablet_manager.cc:1428] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:47:12.592729 20275 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.592828 20275 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 05:47:12.606554 19960 tablet.cc:2369] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:12.918437 20277 raft_consensus.cc:482] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:12.918555 20277 raft_consensus.cc:504] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.918776 20277 leader_election.cc:288] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113), abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769)
I20260926 05:47:12.919032 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113b15f7f1a7456ea778217327b4acf4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9" is_pre_election: true
I20260926 05:47:12.919081 20193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113b15f7f1a7456ea778217327b4acf4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" is_pre_election: true
I20260926 05:47:12.919157 20053 raft_consensus.cc:2454] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 0.
I20260926 05:47:12.919216 20193 raft_consensus.cc:2454] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 0.
I20260926 05:47:12.919355 19845 leader_election.cc:302] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, d88c693f46144b878b40a533abff39b9; no voters:
I20260926 05:47:12.919480 20277 raft_consensus.cc:2781] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:12.919528 20277 raft_consensus.cc:482] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:12.919543 20277 raft_consensus.cc:3037] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.920351 20277 raft_consensus.cc:504] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.920521 20277 leader_election.cc:288] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 election: Requested vote from peers d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113), abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769)
I20260926 05:47:12.920765 20193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113b15f7f1a7456ea778217327b4acf4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22"
I20260926 05:47:12.920845 20193 raft_consensus.cc:3037] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.920835 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113b15f7f1a7456ea778217327b4acf4" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9"
I20260926 05:47:12.920929 20053 raft_consensus.cc:3037] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:12.921514 20193 raft_consensus.cc:2454] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 1.
I20260926 05:47:12.921536 20053 raft_consensus.cc:2454] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 1.
I20260926 05:47:12.921707 19844 leader_election.cc:302] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, abc3bc947aeb4d8da7643aa62b0d8d22; no voters:
I20260926 05:47:12.921823 20277 raft_consensus.cc:2781] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:12.921947 20277 raft_consensus.cc:686] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [term 1 LEADER]: Becoming Leader. State: Replica: aae1612a24c84953ab304b995f26b151, State: Running, Role: LEADER
I20260926 05:47:12.922035 20277 consensus_queue.cc:237] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:12.922323 20308 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: TxnStatusTablet state changed. Reason: New leader aae1612a24c84953ab304b995f26b151. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.922423 20308 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:47:12.922631 20310 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:47:12.922757 20310 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:47:12.922694 19755 catalog_manager.cc:5495] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae1612a24c84953ab304b995f26b151 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } health_report { overall_health: UNKNOWN } } }
I20260926 05:47:12.952762 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.950157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36505 (local address 127.0.0.1:49882)
0926 05:47:12.950280 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.950295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.950345 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.950512 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.950513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.950514 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:47:12.950600 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.950603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.951259 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.951261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.951860 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.951862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.952261 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.952263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.952342 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.952361 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.952569 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.952580 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.952623 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.952691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 05:47:12.952754 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.950239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.11.247.1:36505)
0926 05:47:12.950310 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.950313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.950325 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:47:12.950439 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.950440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.950453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.950506 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:12.950655 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.951155 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.951956 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.952177 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.952260 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.952451 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.952497 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.952565 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.952637 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.952661 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:47:12.954067 20193 raft_consensus.cc:1261] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Refusing update from remote peer aae1612a24c84953ab304b995f26b151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:12.954103 20053 raft_consensus.cc:1261] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Refusing update from remote peer aae1612a24c84953ab304b995f26b151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:12.954375 20308 consensus_queue.cc:1035] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:12.954526 20277 consensus_queue.cc:1035] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:12.955122 20275 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: TxnStatusTablet state changed. Reason: New leader aae1612a24c84953ab304b995f26b151. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.955231 20275 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:47:12.955263 20273 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: TxnStatusTablet state changed. Reason: New leader aae1612a24c84953ab304b995f26b151. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.955358 20273 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:47:12.956189 20308 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.956295 20308 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:47:12.956434 20313 mvcc.cc:205] Tried to move back new op lower bound from 7333485088578064384 to 7333485088449110016. Current Snapshot: MvccSnapshot[applied={T|T < 7333485088578064384}]
I20260926 05:47:12.956480 20275 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.956558 20275 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P d88c693f46144b878b40a533abff39b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:47:12.956521 20308 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.956593 20308 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P aae1612a24c84953ab304b995f26b151: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:47:12.956921 20273 tablet_replica.cc:439] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } } }
I20260926 05:47:12.957026 20273 tablet_replica.cc:442] T 113b15f7f1a7456ea778217327b4acf4 P abc3bc947aeb4d8da7643aa62b0d8d22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:47:12.961793 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.959036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.11.247.3:38769)
0926 05:47:12.959147 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.959150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.959157 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:12.959215 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.959215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.959238 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.959342 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:12.959541 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.960035 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.960906 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.961132 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.961211 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.961393 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.961527 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.961611 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.961664 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.961691 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":46,"threads_started":1}
I20260926 05:47:12.967329 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.964619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36505 (local address 127.11.247.3:45857)
0926 05:47:12.964840 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.964849 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.964924 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.965063 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.965064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.965066 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:47:12.965123 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.965125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.965778 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.965779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.966382 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.966384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.966851 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.966852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.966924 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.966940 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.967151 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.967153 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.967197 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.967248 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
I20260926 05:47:12.967306 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.964691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45857 (local address 127.11.247.1:36505)
0926 05:47:12.964764 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.964767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.964908 (+ 141us) server_negotiation.cc:407] Connection header received
0926 05:47:12.964988 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.964989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.965004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.965059 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:12.965188 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.965677 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.966488 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.966769 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.966849 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.967046 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.967085 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.967137 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.967198 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.967221 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:47:12.970178 20053 raft_consensus.cc:1261] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Refusing update from remote peer abc3bc947aeb4d8da7643aa62b0d8d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:12.970329 19894 raft_consensus.cc:1261] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Refusing update from remote peer abc3bc947aeb4d8da7643aa62b0d8d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:12.970494 20297 consensus_queue.cc:1035] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:12.970656 20273 consensus_queue.cc:1035] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:12.981812 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.977352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:40267 (local address 127.11.247.3:38769)
0926 05:47:12.977394 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:12.977396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:12.978883 (+ 1487us) server_negotiation.cc:407] Connection header received
0926 05:47:12.978920 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:12.978921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:12.978936 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:12.978989 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:12.979202 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.979982 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.980858 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.981188 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.981259 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.981526 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.981567 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.981594 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:12.981694 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 05:47:12.981721 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:47:12.982265 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:12.977288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:38769 (local address 127.11.247.1:40267)
0926 05:47:12.978740 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:12.978754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:12.978829 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:12.979049 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:12.979050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:12.979052 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:47:12.979134 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.979137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.980122 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.980123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.980760 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:12.980763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:12.981301 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:12.981303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:12.981377 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:12.981405 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:47:12.981633 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:47:12.981634 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:47:12.982115 (+ 481us) client_negotiation.cc:240] Negotiation successful
0926 05:47:12.982178 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"thread_start_us":85,"threads_started":1}
I20260926 05:47:12.990068 19755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:12.990191 19755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:12.991074 19755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Sending DeleteTablet for 3 replicas of tablet b2c6f8df6d154701ba2079645d80c402
I20260926 05:47:12.991338 20173 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c6f8df6d154701ba2079645d80c402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:12 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45134
I20260926 05:47:12.991535 20033 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c6f8df6d154701ba2079645d80c402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:12 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:58100
I20260926 05:47:12.991554 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c6f8df6d154701ba2079645d80c402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:12 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:49862
I20260926 05:47:12.991582 20336 tablet_replica.cc:330] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: stopping tablet replica
I20260926 05:47:12.991658 20336 raft_consensus.cc:2229] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:12.991766 20336 raft_consensus.cc:2258] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:12.992033 20336 ts_tablet_manager.cc:1905] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:12.994139 20336 ts_tablet_manager.cc:1918] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:47:12.994210 20336 log.cc:1196] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/b2c6f8df6d154701ba2079645d80c402
I20260926 05:47:12.994436 20336 ts_tablet_manager.cc:1939] T b2c6f8df6d154701ba2079645d80c402 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting consensus metadata
I20260926 05:47:12.994871 19740 catalog_manager.cc:4841] TS abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769): tablet b2c6f8df6d154701ba2079645d80c402 (table test-table [id=c4036be955854dabb5a2b27aac00c671]) successfully deleted
I20260926 05:47:12.992141 20338 tablet_replica.cc:330] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: stopping tablet replica
I20260926 05:47:12.995301 20338 raft_consensus.cc:2229] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:12.995404 20338 raft_consensus.cc:2258] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:12.996855 20338 ts_tablet_manager.cc:1905] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:12.998692 20338 ts_tablet_manager.cc:1918] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:47:12.998819 20338 log.cc:1196] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/b2c6f8df6d154701ba2079645d80c402
I20260926 05:47:12.999099 20338 ts_tablet_manager.cc:1939] T b2c6f8df6d154701ba2079645d80c402 P d88c693f46144b878b40a533abff39b9: Deleting consensus metadata
I20260926 05:47:12.999532 19741 catalog_manager.cc:4841] TS d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113): tablet b2c6f8df6d154701ba2079645d80c402 (table test-table [id=c4036be955854dabb5a2b27aac00c671]) successfully deleted
I20260926 05:47:13.002734 20337 tablet_replica.cc:330] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: stopping tablet replica
I20260926 05:47:13.002820 20337 raft_consensus.cc:2229] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:13.002871 20337 raft_consensus.cc:2258] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:13.003532 20337 ts_tablet_manager.cc:1905] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:13.005780 20337 ts_tablet_manager.cc:1918] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:47:13.005927 20337 log.cc:1196] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/b2c6f8df6d154701ba2079645d80c402
I20260926 05:47:13.006211 20337 ts_tablet_manager.cc:1939] T b2c6f8df6d154701ba2079645d80c402 P aae1612a24c84953ab304b995f26b151: Deleting consensus metadata
I20260926 05:47:13.006671 19742 catalog_manager.cc:4841] TS aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505): tablet b2c6f8df6d154701ba2079645d80c402 (table test-table [id=c4036be955854dabb5a2b27aac00c671]) successfully deleted
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:47:13.029551 19805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:13.022249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.11.247.62:45197)
0926 05:47:13.022307 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:13.022310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:13.022457 (+ 147us) server_negotiation.cc:407] Connection header received
0926 05:47:13.022520 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:13.022521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:13.022536 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:13.022635 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:13.023460 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.024262 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.025385 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.025701 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.025775 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:13.029473 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40340: BlockingRecv error: recv got EOF from 127.0.0.1:40340 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM
I20260926 05:47:13.040182 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:13.030707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.11.247.62:45197)
0926 05:47:13.030750 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:13.030752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:13.031036 (+ 284us) server_negotiation.cc:407] Connection header received
0926 05:47:13.031076 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:13.031077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:13.031090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:13.031157 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:13.031749 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.032540 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.033655 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.033941 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.034026 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:13.036944 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:13.036946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:13.036946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:13.036957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:13.038773 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:13.039233 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:13.039235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:13.039235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:13.039297 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:13.039532 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:13.039533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:13.039533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:13.039684 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:13.039768 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:13.039984 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 05:47:13.040089 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:47:13.044653 19755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:47:13.044865 19755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:13.046379 19891 tablet_service.cc:1467] Processing CreateTablet for tablet f1564f65fdbe4c2c9dee9962e0991435 (DEFAULT_TABLE table=test-table [id=e07cb3525eb84cedb290069e285d0230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:13.046368 20173 tablet_service.cc:1467] Processing CreateTablet for tablet f1564f65fdbe4c2c9dee9962e0991435 (DEFAULT_TABLE table=test-table [id=e07cb3525eb84cedb290069e285d0230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:13.046531 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1564f65fdbe4c2c9dee9962e0991435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:13.046531 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1564f65fdbe4c2c9dee9962e0991435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:13.047912 20268 tablet_bootstrap.cc:492] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Bootstrap starting.
I20260926 05:47:13.048290 20033 tablet_service.cc:1467] Processing CreateTablet for tablet f1564f65fdbe4c2c9dee9962e0991435 (DEFAULT_TABLE table=test-table [id=e07cb3525eb84cedb290069e285d0230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:13.048411 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1564f65fdbe4c2c9dee9962e0991435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:13.048451 20268 tablet_bootstrap.cc:654] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:13.048863 20267 tablet_bootstrap.cc:492] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Bootstrap starting.
I20260926 05:47:13.049329 20267 tablet_bootstrap.cc:654] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:13.049744 20271 tablet_bootstrap.cc:492] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Bootstrap starting.
I20260926 05:47:13.050069 20271 tablet_bootstrap.cc:654] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:13.050434 20268 tablet_bootstrap.cc:492] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: No bootstrap required, opened a new log
I20260926 05:47:13.050493 20268 ts_tablet_manager.cc:1397] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 05:47:13.050719 20271 tablet_bootstrap.cc:492] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: No bootstrap required, opened a new log
I20260926 05:47:13.050683 20268 raft_consensus.cc:348] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.050752 20271 ts_tablet_manager.cc:1397] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:13.050765 20268 raft_consensus.cc:374] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:13.050788 20268 raft_consensus.cc:729] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1612a24c84953ab304b995f26b151, State: Initialized, Role: FOLLOWER
I20260926 05:47:13.050846 20268 consensus_queue.cc:260] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.050899 20271 raft_consensus.cc:348] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.050952 20268 ts_tablet_manager.cc:1428] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:13.050971 20271 raft_consensus.cc:374] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:13.050990 20271 raft_consensus.cc:729] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88c693f46144b878b40a533abff39b9, State: Initialized, Role: FOLLOWER
I20260926 05:47:13.051029 20271 consensus_queue.cc:260] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.051132 20271 ts_tablet_manager.cc:1428] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:13.055229 20267 tablet_bootstrap.cc:492] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: No bootstrap required, opened a new log
I20260926 05:47:13.055316 20267 ts_tablet_manager.cc:1397] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260926 05:47:13.055491 20267 raft_consensus.cc:348] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.055578 20267 raft_consensus.cc:374] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:13.055600 20267 raft_consensus.cc:729] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Initialized, Role: FOLLOWER
I20260926 05:47:13.055655 20267 consensus_queue.cc:260] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.055761 20267 ts_tablet_manager.cc:1428] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:13.076014 20277 raft_consensus.cc:482] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:13.076148 20277 raft_consensus.cc:504] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.076319 20277 leader_election.cc:288] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113), abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769)
I20260926 05:47:13.076579 20193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1564f65fdbe4c2c9dee9962e0991435" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" is_pre_election: true
I20260926 05:47:13.076593 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1564f65fdbe4c2c9dee9962e0991435" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9" is_pre_election: true
I20260926 05:47:13.076668 20193 raft_consensus.cc:2454] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 0.
I20260926 05:47:13.076707 20053 raft_consensus.cc:2454] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 0.
I20260926 05:47:13.076823 19844 leader_election.cc:302] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, abc3bc947aeb4d8da7643aa62b0d8d22; no voters:
I20260926 05:47:13.076921 20277 raft_consensus.cc:2781] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:13.076954 20277 raft_consensus.cc:482] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:13.076972 20277 raft_consensus.cc:3037] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:13.077885 20277 raft_consensus.cc:504] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.078047 20277 leader_election.cc:288] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 election: Requested vote from peers d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113), abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769)
I20260926 05:47:13.078235 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1564f65fdbe4c2c9dee9962e0991435" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9"
I20260926 05:47:13.078260 20193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1564f65fdbe4c2c9dee9962e0991435" candidate_uuid: "aae1612a24c84953ab304b995f26b151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22"
I20260926 05:47:13.078315 20053 raft_consensus.cc:3037] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:13.078332 20193 raft_consensus.cc:3037] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:13.079140 20053 raft_consensus.cc:2454] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 1.
I20260926 05:47:13.079183 20193 raft_consensus.cc:2454] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae1612a24c84953ab304b995f26b151 in term 1.
I20260926 05:47:13.079335 19845 leader_election.cc:302] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, d88c693f46144b878b40a533abff39b9; no voters:
I20260926 05:47:13.079602 20277 raft_consensus.cc:2781] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:13.079668 20277 raft_consensus.cc:686] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 1 LEADER]: Becoming Leader. State: Replica: aae1612a24c84953ab304b995f26b151, State: Running, Role: LEADER
I20260926 05:47:13.079713 20277 consensus_queue.cc:237] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.080368 19756 catalog_manager.cc:5495] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae1612a24c84953ab304b995f26b151 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "aae1612a24c84953ab304b995f26b151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } health_report { overall_health: UNKNOWN } } }
I20260926 05:47:13.093513 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:13.090550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.11.247.1:36505)
0926 05:47:13.090650 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:13.090652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:13.090663 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:13.090693 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:13.090694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:13.090707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:13.090789 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:13.091068 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.091552 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.092771 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.092966 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.093048 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:13.093227 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:13.093259 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:13.093300 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:13.093381 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 05:47:13.093400 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 05:47:13.094425 20053 raft_consensus.cc:1261] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Refusing update from remote peer aae1612a24c84953ab304b995f26b151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:13.094439 20193 raft_consensus.cc:1261] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Refusing update from remote peer aae1612a24c84953ab304b995f26b151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:13.094740 20277 consensus_queue.cc:1035] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:13.094898 20320 consensus_queue.cc:1035] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:13.098059 19755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:13.098174 19755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:13.099457 19755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Sending DeleteTablet for 3 replicas of tablet f1564f65fdbe4c2c9dee9962e0991435
I20260926 05:47:13.100386 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet f1564f65fdbe4c2c9dee9962e0991435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:13 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:49862
I20260926 05:47:13.100548 20337 tablet_replica.cc:330] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: stopping tablet replica
I20260926 05:47:13.100529 20033 tablet_service.cc:1514] Processing DeleteTablet for tablet f1564f65fdbe4c2c9dee9962e0991435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:13 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:58100
I20260926 05:47:13.100620 20337 raft_consensus.cc:2229] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:13.100697 20337 raft_consensus.cc:2258] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:13.100739 20338 tablet_replica.cc:330] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: stopping tablet replica
I20260926 05:47:13.100812 20338 raft_consensus.cc:2229] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:13.100862 20338 raft_consensus.cc:2258] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:13.100953 20173 tablet_service.cc:1514] Processing DeleteTablet for tablet f1564f65fdbe4c2c9dee9962e0991435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:13 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45134
I20260926 05:47:13.101199 20336 tablet_replica.cc:330] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: stopping tablet replica
I20260926 05:47:13.101274 20336 raft_consensus.cc:2229] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:13.101178 19755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:47:13.101333 20336 raft_consensus.cc:2258] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 05:47:13.101423 19755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:13.101984 20336 ts_tablet_manager.cc:1905] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:13.102139 20337 ts_tablet_manager.cc:1905] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:13.103018 20173 tablet_service.cc:1467] Processing CreateTablet for tablet c89cf5ae8de8446c98b0e49a874e3dbd (DEFAULT_TABLE table=test-table [id=3fb31c55f9f84a6a97a6fdac82af101a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:13.103139 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89cf5ae8de8446c98b0e49a874e3dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:13.103866 20337 ts_tablet_manager.cc:1918] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:13.103865 20336 ts_tablet_manager.cc:1918] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:13.103927 20336 log.cc:1196] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/f1564f65fdbe4c2c9dee9962e0991435
I20260926 05:47:13.103927 20337 log.cc:1196] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/f1564f65fdbe4c2c9dee9962e0991435
I20260926 05:47:13.104185 20336 ts_tablet_manager.cc:1939] T f1564f65fdbe4c2c9dee9962e0991435 P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting consensus metadata
I20260926 05:47:13.104192 20337 ts_tablet_manager.cc:1939] T f1564f65fdbe4c2c9dee9962e0991435 P aae1612a24c84953ab304b995f26b151: Deleting consensus metadata
I20260926 05:47:13.104588 20267 tablet_bootstrap.cc:492] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Bootstrap starting.
I20260926 05:47:13.104620 20338 ts_tablet_manager.cc:1905] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:13.104784 19742 catalog_manager.cc:4841] TS aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505): tablet f1564f65fdbe4c2c9dee9962e0991435 (table test-table [id=e07cb3525eb84cedb290069e285d0230]) successfully deleted
I20260926 05:47:13.105026 20267 tablet_bootstrap.cc:654] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:13.105727 20267 tablet_bootstrap.cc:492] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: No bootstrap required, opened a new log
I20260926 05:47:13.105772 20267 ts_tablet_manager.cc:1397] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:13.105788 20033 tablet_service.cc:1467] Processing CreateTablet for tablet c89cf5ae8de8446c98b0e49a874e3dbd (DEFAULT_TABLE table=test-table [id=3fb31c55f9f84a6a97a6fdac82af101a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:13.105891 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89cf5ae8de8446c98b0e49a874e3dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:13.105919 20267 raft_consensus.cc:348] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.106001 20267 raft_consensus.cc:374] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:13.106024 20267 raft_consensus.cc:729] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Initialized, Role: FOLLOWER
I20260926 05:47:13.106091 20267 consensus_queue.cc:260] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.106205 20267 ts_tablet_manager.cc:1428] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:13.106218 20338 ts_tablet_manager.cc:1918] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:13.106273 20338 log.cc:1196] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/f1564f65fdbe4c2c9dee9962e0991435
I20260926 05:47:13.106499 20338 ts_tablet_manager.cc:1939] T f1564f65fdbe4c2c9dee9962e0991435 P d88c693f46144b878b40a533abff39b9: Deleting consensus metadata
I20260926 05:47:13.106892 19891 tablet_service.cc:1467] Processing CreateTablet for tablet c89cf5ae8de8446c98b0e49a874e3dbd (DEFAULT_TABLE table=test-table [id=3fb31c55f9f84a6a97a6fdac82af101a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:13.107024 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89cf5ae8de8446c98b0e49a874e3dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:13.107131 20271 tablet_bootstrap.cc:492] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Bootstrap starting.
I20260926 05:47:13.107851 20271 tablet_bootstrap.cc:654] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:13.108183 20268 tablet_bootstrap.cc:492] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151: Bootstrap starting.
I20260926 05:47:13.108322 19741 catalog_manager.cc:4841] TS d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113): tablet f1564f65fdbe4c2c9dee9962e0991435 (table test-table [id=e07cb3525eb84cedb290069e285d0230]) successfully deleted
I20260926 05:47:13.108520 20268 tablet_bootstrap.cc:654] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:13.109055 20271 tablet_bootstrap.cc:492] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: No bootstrap required, opened a new log
I20260926 05:47:13.109107 20271 ts_tablet_manager.cc:1397] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:47:13.109148 19740 catalog_manager.cc:4841] TS abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769): tablet f1564f65fdbe4c2c9dee9962e0991435 (table test-table [id=e07cb3525eb84cedb290069e285d0230]) successfully deleted
I20260926 05:47:13.109254 20271 raft_consensus.cc:348] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.109340 20271 raft_consensus.cc:374] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:13.109362 20271 raft_consensus.cc:729] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88c693f46144b878b40a533abff39b9, State: Initialized, Role: FOLLOWER
I20260926 05:47:13.109404 20271 consensus_queue.cc:260] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.109498 20271 ts_tablet_manager.cc:1428] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:13.110265 20268 tablet_bootstrap.cc:492] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151: No bootstrap required, opened a new log
I20260926 05:47:13.110327 20268 ts_tablet_manager.cc:1397] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:47:13.110438 20268 raft_consensus.cc:348] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.110522 20268 raft_consensus.cc:374] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:13.110544 20268 raft_consensus.cc:729] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae1612a24c84953ab304b995f26b151, State: Initialized, Role: FOLLOWER
I20260926 05:47:13.110647 20268 consensus_queue.cc:260] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.110771 20268 ts_tablet_manager.cc:1428] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:47:13.274827 20333 raft_consensus.cc:482] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:13.274955 20333 raft_consensus.cc:504] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.275170 20333 leader_election.cc:288] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505), d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:13.275467 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cf5ae8de8446c98b0e49a874e3dbd" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9" is_pre_election: true
I20260926 05:47:13.275467 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cf5ae8de8446c98b0e49a874e3dbd" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae1612a24c84953ab304b995f26b151" is_pre_election: true
I20260926 05:47:13.275585 19894 raft_consensus.cc:2454] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 0.
I20260926 05:47:13.275585 20053 raft_consensus.cc:2454] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 0.
I20260926 05:47:13.275815 20126 leader_election.cc:302] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae1612a24c84953ab304b995f26b151, abc3bc947aeb4d8da7643aa62b0d8d22; no voters:
I20260926 05:47:13.275924 20333 raft_consensus.cc:2781] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:13.276003 20333 raft_consensus.cc:482] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:13.276047 20333 raft_consensus.cc:3037] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:13.276837 20333 raft_consensus.cc:504] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.277062 20333 leader_election.cc:288] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 election: Requested vote from peers aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505), d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113)
I20260926 05:47:13.277218 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cf5ae8de8446c98b0e49a874e3dbd" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88c693f46144b878b40a533abff39b9"
I20260926 05:47:13.277217 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cf5ae8de8446c98b0e49a874e3dbd" candidate_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae1612a24c84953ab304b995f26b151"
I20260926 05:47:13.277309 19894 raft_consensus.cc:3037] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:13.277313 20053 raft_consensus.cc:3037] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:13.278126 20053 raft_consensus.cc:2454] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 1.
I20260926 05:47:13.278143 19894 raft_consensus.cc:2454] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc3bc947aeb4d8da7643aa62b0d8d22 in term 1.
I20260926 05:47:13.278337 20125 leader_election.cc:302] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc3bc947aeb4d8da7643aa62b0d8d22, d88c693f46144b878b40a533abff39b9; no voters:
I20260926 05:47:13.278448 20333 raft_consensus.cc:2781] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:13.278502 20333 raft_consensus.cc:686] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 LEADER]: Becoming Leader. State: Replica: abc3bc947aeb4d8da7643aa62b0d8d22, State: Running, Role: LEADER
I20260926 05:47:13.278564 20333 consensus_queue.cc:237] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } }
I20260926 05:47:13.279268 19755 catalog_manager.cc:5495] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc3bc947aeb4d8da7643aa62b0d8d22" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38769 } health_report { overall_health: HEALTHY } } }
I20260926 05:47:13.300397 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:13.297999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.11.247.3:38769)
0926 05:47:13.298056 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:13.298058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:13.298093 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:47:13.298120 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:13.298121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:13.298132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:13.298185 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:13.298273 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.298821 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.299649 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:13.299859 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:13.299924 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:13.300090 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:13.300119 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:13.300160 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:13.300263 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 05:47:13.300283 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:47:13.302858 19894 raft_consensus.cc:1261] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Refusing update from remote peer abc3bc947aeb4d8da7643aa62b0d8d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:13.302891 20053 raft_consensus.cc:1261] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Refusing update from remote peer abc3bc947aeb4d8da7643aa62b0d8d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:13.303149 20273 consensus_queue.cc:1035] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88c693f46144b878b40a533abff39b9" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:13.303248 20333 consensus_queue.cc:1035] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae1612a24c84953ab304b995f26b151" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:13.313362 19755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:13.313467 19755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:13.314162 19755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd066e1173da44bb8a970ef05ad3c863: Sending DeleteTablet for 3 replicas of tablet c89cf5ae8de8446c98b0e49a874e3dbd
I20260926 05:47:13.314420 20033 tablet_service.cc:1514] Processing DeleteTablet for tablet c89cf5ae8de8446c98b0e49a874e3dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:13 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:58100
I20260926 05:47:13.314438 20173 tablet_service.cc:1514] Processing DeleteTablet for tablet c89cf5ae8de8446c98b0e49a874e3dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:13 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45134
I20260926 05:47:13.314428 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet c89cf5ae8de8446c98b0e49a874e3dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:13 UTC) from {username='oryx', principal='oryx/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:49862
I20260926 05:47:13.314527 20336 tablet_replica.cc:330] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: stopping tablet replica
I20260926 05:47:13.314610 20338 tablet_replica.cc:330] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: stopping tablet replica
I20260926 05:47:13.314634 20336 raft_consensus.cc:2229] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:13.314643 20337 tablet_replica.cc:330] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151: stopping tablet replica
I20260926 05:47:13.314677 20338 raft_consensus.cc:2229] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:13.314689 20336 raft_consensus.cc:2258] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:13.314694 20337 raft_consensus.cc:2229] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:13.314711 20338 raft_consensus.cc:2258] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:13.314728 20337 raft_consensus.cc:2258] T c89cf5ae8de8446c98b0e49a874e3dbd P aae1612a24c84953ab304b995f26b151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:13.315034 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 19828
I20260926 05:47:13.315032 20338 ts_tablet_manager.cc:1905] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:13.316841 20338 ts_tablet_manager.cc:1918] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:47:13.316905 20338 log.cc:1196] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/c89cf5ae8de8446c98b0e49a874e3dbd
I20260926 05:47:13.317124 20338 ts_tablet_manager.cc:1939] T c89cf5ae8de8446c98b0e49a874e3dbd P d88c693f46144b878b40a533abff39b9: Deleting consensus metadata
I20260926 05:47:13.317616 19741 catalog_manager.cc:4841] TS d88c693f46144b878b40a533abff39b9 (127.11.247.2:34113): tablet c89cf5ae8de8446c98b0e49a874e3dbd (table test-table [id=3fb31c55f9f84a6a97a6fdac82af101a]) successfully deleted
I20260926 05:47:13.317762 20336 ts_tablet_manager.cc:1905] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:13.319255 20336 ts_tablet_manager.cc:1918] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:47:13.319309 20336 log.cc:1196] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/c89cf5ae8de8446c98b0e49a874e3dbd
I20260926 05:47:13.319526 20336 ts_tablet_manager.cc:1939] T c89cf5ae8de8446c98b0e49a874e3dbd P abc3bc947aeb4d8da7643aa62b0d8d22: Deleting consensus metadata
I20260926 05:47:13.319865 19740 catalog_manager.cc:4841] TS abc3bc947aeb4d8da7643aa62b0d8d22 (127.11.247.3:38769): tablet c89cf5ae8de8446c98b0e49a874e3dbd (table test-table [id=3fb31c55f9f84a6a97a6fdac82af101a]) successfully deleted
W20260926 05:47:13.323293 19742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:36505 (error 108)
I20260926 05:47:13.323493 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 19968
I20260926 05:47:13.324785 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:13.324565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36505 (local address 127.0.0.1:49900)
0926 05:47:13.324676 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:13.324732 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
W20260926 05:47:13.328611 19742 catalog_manager.cc:4584] TS aae1612a24c84953ab304b995f26b151 (127.11.247.1:36505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c89cf5ae8de8446c98b0e49a874e3dbd: Network error: Client connection negotiation failed: client connection to 127.11.247.1:36505: connect: Connection refused (error 111)
I20260926 05:47:13.328752 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20108
I20260926 05:47:13.334789 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 19724
2026-09-26T05:47:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3023 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[20368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[20368](info): set up 2 sockets
Sep 26 05:47:13 dist-test-slave-21p4 krb5kdc[20368](info): commencing operation
krb5kdc: starting...
W20260926 05:47:14.379225 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:14 dist-test-slave-21p4 krb5kdc[20368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:47:14.505285 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:36977
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:36977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:14.591763 20381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:14.592083 20381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:14.592135 20381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:14.595002 20381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:14.595093 20381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:14.595125 20381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:14.595165 20381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:14.595193 20381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:14.598827 20381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:36977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:14.599191 20381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:14.599465 20381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:14.606933 20387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:14.607213 20381 server_base.cc:1034] running on GCE node
W20260926 05:47:14.606930 20386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:14.606937 20389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:14.607620 20381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:14.608021 20381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:14.609225 20381 hybrid_clock.cc:648] HybridClock initialized: now 1790401634609213 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T05:47:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1277 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:14 dist-test-slave-21p4 krb5kdc[20394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:14 dist-test-slave-21p4 krb5kdc[20394](info): set up 2 sockets
Sep 26 05:47:14 dist-test-slave-21p4 krb5kdc[20394](info): commencing operation
krb5kdc: starting...
W20260926 05:47:15.648156 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:15 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:47:15.778676 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:41141
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:41141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:15.865245 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:15.865477 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:15.865501 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:15.866851 20407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:15.866905 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:15.866922 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:15.866936 20407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:15.866948 20407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:15.868533 20407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:41141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:15.868800 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:15.869045 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:15.874409 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:15.874413 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:15.874437 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:15.874692 20407 server_base.cc:1034] running on GCE node
I20260926 05:47:15.874972 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:15.875308 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:15.876473 20407 hybrid_clock.cc:648] HybridClock initialized: now 1790401635876449 us; error 37 us; skew 500 ppm
Sep 26 05:47:15 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401635, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:15.884284 20407 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:47:15.884843 20407 webserver.cc:460] Webserver started at http://127.11.247.62:35201/ using document root <none> and password file <none>
I20260926 05:47:15.885092 20407 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:15.885164 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:15.885351 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:15.886839 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "bf7fb445098a4ed3a9cfb939df0cca48"
format_stamp: "Formatted at 2026-09-26 05:47:15 on dist-test-slave-21p4"
server_key: "8eabdd31f3b8685f1f9963fddfd40b99"
server_key_iv: "ff7b439cd53ed0216093121cf32b7beb"
server_key_version: "encryptionkey@0"
I20260926 05:47:15.887329 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "bf7fb445098a4ed3a9cfb939df0cca48"
format_stamp: "Formatted at 2026-09-26 05:47:15 on dist-test-slave-21p4"
server_key: "8eabdd31f3b8685f1f9963fddfd40b99"
server_key_iv: "ff7b439cd53ed0216093121cf32b7beb"
server_key_version: "encryptionkey@0"
I20260926 05:47:15.888840 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:15.889694 20422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:15.889910 20407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:15.890027 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "bf7fb445098a4ed3a9cfb939df0cca48"
format_stamp: "Formatted at 2026-09-26 05:47:15 on dist-test-slave-21p4"
server_key: "8eabdd31f3b8685f1f9963fddfd40b99"
server_key_iv: "ff7b439cd53ed0216093121cf32b7beb"
server_key_version: "encryptionkey@0"
I20260926 05:47:15.890141 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:15.903192 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:15.905534 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:15.905735 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:15.910178 20407 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:41141
I20260926 05:47:15.910212 20474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:41141 every 8 connection(s)
I20260926 05:47:15.910693 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:15.911576 20475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:15.913857 20475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Bootstrap starting.
I20260926 05:47:15.914032 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20407
I20260926 05:47:15.914150 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:15.914378 12252 external_mini_cluster.cc:1428] Setting key a481f71bd992427535b349d7f5fe21b3
I20260926 05:47:15.914660 20475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:15.915015 20475 log.cc:824] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:15.915807 20475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: No bootstrap required, opened a new log
I20260926 05:47:15.918223 20475 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:15.918412 20475 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:15.918447 20475 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fb445098a4ed3a9cfb939df0cca48, State: Initialized, Role: FOLLOWER
I20260926 05:47:15.918565 20475 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:15.918661 20475 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:15.918694 20475 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:15.918763 20475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:15.919387 20475 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:15.919507 20475 leader_election.cc:302] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7fb445098a4ed3a9cfb939df0cca48; no voters:
I20260926 05:47:15.919693 20475 leader_election.cc:288] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:15.919741 20480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:15.919859 20480 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 LEADER]: Becoming Leader. State: Replica: bf7fb445098a4ed3a9cfb939df0cca48, State: Running, Role: LEADER
I20260926 05:47:15.919989 20475 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:15.919974 20480 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
Sep 26 05:47:15 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:15.921689 20481 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7fb445098a4ed3a9cfb939df0cca48. Latest consensus state: current_term: 1 leader_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } } }
I20260926 05:47:15.921784 20481 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:15.922098 20482 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } } }
I20260926 05:47:15.922169 20482 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:15.922565 20489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:15.923214 20489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:15.924993 20489 catalog_manager.cc:1347] Generated new cluster ID: a485b2563744463e89d398c0a72c8e34
I20260926 05:47:15.925065 20489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:15.925886 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:15.914968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.11.247.62:41141)
0926 05:47:15.915252 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:15.915262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:15.915297 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:47:15.915838 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:15.915846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:15.915920 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:15.916072 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:15.916871 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:15.917774 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:15.918892 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:15.919318 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:15.919400 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:15.922129 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:15.922133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:15.922133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:15.922145 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:15.924192 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:15.924605 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:15.924608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:15.924609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:15.924715 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:15.924933 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:15.924934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:15.924935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:15.925237 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:15.925387 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:15.925526 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:47:15.925684 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
I20260926 05:47:15.943912 20489 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:15.944729 20489 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:15.948261 20489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Generated new TSK 0
I20260926 05:47:15.948462 20489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:47:15.982126 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41141
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:16.066241 20501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:16.066466 20501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:16.066493 20501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:16.067813 20501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:16.067862 20501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:16.067891 20501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:16.069388 20501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:16.069659 20501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:16.069911 20501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:16.075352 20509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.075369 20507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.075363 20506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.075904 20501 server_base.cc:1034] running on GCE node
I20260926 05:47:16.076129 20501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:16.076432 20501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:16.077600 20501 hybrid_clock.cc:648] HybridClock initialized: now 1790401636077582 us; error 36 us; skew 500 ppm
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:16.080318 20501 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:47:16.080789 20501 webserver.cc:460] Webserver started at http://127.11.247.1:37859/ using document root <none> and password file <none>
I20260926 05:47:16.081046 20501 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:16.081116 20501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:16.081260 20501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:16.082350 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "d91395b812844b4ca3d625f964d49319"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "dc9681220ff722e1b52f2a0ca65d7999"
server_key_iv: "cd5d1d217c7a29d5caeeb6aa2eb00986"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.082741 20501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "d91395b812844b4ca3d625f964d49319"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "dc9681220ff722e1b52f2a0ca65d7999"
server_key_iv: "cd5d1d217c7a29d5caeeb6aa2eb00986"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.084141 20501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:16.084995 20516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.085183 20501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:16.085289 20501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "d91395b812844b4ca3d625f964d49319"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "dc9681220ff722e1b52f2a0ca65d7999"
server_key_iv: "cd5d1d217c7a29d5caeeb6aa2eb00986"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.085366 20501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:16.097332 20501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:16.100145 20501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:16.100358 20501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:16.100625 20501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:16.100961 20501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:16.101014 20501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.101056 20501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:16.101087 20501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.107229 20501 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:45383
I20260926 05:47:16.107276 20629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:45383 every 8 connection(s)
I20260926 05:47:16.107657 20501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:16.117827 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20501
I20260926 05:47:16.117956 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:16.118168 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.108510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:44973 (local address 127.11.247.62:41141)
0926 05:47:16.108594 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.108597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.108822 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:47:16.109311 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:16.109313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:16.109322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:16.109377 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:16.110408 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.111016 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.112075 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.112288 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.112353 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.115054 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:16.115080 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:16.115080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:16.115089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:16.116654 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:16.117105 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:16.117106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:16.117107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:16.117160 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:16.117395 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:16.117395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:16.117395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:16.117509 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:16.117560 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:16.117875 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 05:47:16.118002 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:47:16.118189 12252 external_mini_cluster.cc:1428] Setting key f6bcab0825dd08cb9f0500268c7753b3
I20260926 05:47:16.118328 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.108464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:44973)
0926 05:47:16.108684 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.108733 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:16.109228 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:16.109479 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:16.109483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:16.110076 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:16.110295 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.110300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.111119 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.111120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.111978 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.111980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.112426 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.112428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.112584 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.113198 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:16.113207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:16.114921 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:16.116749 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:16.116758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:16.116760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:16.117019 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:16.117221 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:16.117222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:16.117222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:16.117338 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:16.117581 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:16.117586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:16.117740 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 05:47:16.117828 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:47:16.118055 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
I20260926 05:47:16.118940 20630 heartbeater.cc:347] Connected to a master server at 127.11.247.62:41141
I20260926 05:47:16.119081 20630 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:16.119274 20630 heartbeater.cc:511] Master 127.11.247.62:41141 requested a full tablet report, sending...
I20260926 05:47:16.119899 20439 ts_manager.cc:194] Registered new tserver with Master: d91395b812844b4ca3d625f964d49319 (127.11.247.1:45383)
I20260926 05:47:16.120774 20439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:44973
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:47:16.147380 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41141
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:16.229768 20635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:16.230038 20635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:16.230082 20635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:16.231334 20635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:16.231406 20635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:16.231451 20635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:16.232945 20635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:16.233223 20635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:16.233465 20635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:47:16.238977 20635 server_base.cc:1034] running on GCE node
W20260926 05:47:16.238967 20641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.238967 20643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.239013 20640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.239547 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:16.239828 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:16.241010 20635 hybrid_clock.cc:648] HybridClock initialized: now 1790401636240996 us; error 56 us; skew 500 ppm
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:16.243737 20635 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:47:16.244234 20635 webserver.cc:460] Webserver started at http://127.11.247.2:32827/ using document root <none> and password file <none>
I20260926 05:47:16.244477 20635 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:16.244546 20635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:16.244704 20635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:16.245826 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "56e56ab1b1134e918acd5dbce26e9873"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "6b5ea9e773507502fa41226d60fb2c00"
server_key_iv: "212bfa1411063fdbcfd609c4d1bcc402"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.246217 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "56e56ab1b1134e918acd5dbce26e9873"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "6b5ea9e773507502fa41226d60fb2c00"
server_key_iv: "212bfa1411063fdbcfd609c4d1bcc402"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.247720 20635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:47:16.248603 20650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.248862 20635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.248988 20635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "56e56ab1b1134e918acd5dbce26e9873"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "6b5ea9e773507502fa41226d60fb2c00"
server_key_iv: "212bfa1411063fdbcfd609c4d1bcc402"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.249078 20635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:16.267781 20635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:16.270617 20635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:16.270825 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:16.271106 20635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:16.271538 20635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:16.271595 20635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.271628 20635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:16.271646 20635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.278028 20635 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:34111
I20260926 05:47:16.278071 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:34111 every 8 connection(s)
I20260926 05:47:16.278424 20635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:47:16.282692 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20635
I20260926 05:47:16.282831 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:16.283038 12252 external_mini_cluster.cc:1428] Setting key 417483cd597a5f28d06b08474ad1062a
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:16.289501 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.279266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:55027 (local address 127.11.247.62:41141)
0926 05:47:16.279328 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.279330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.279510 (+ 180us) server_negotiation.cc:407] Connection header received
0926 05:47:16.280083 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:16.280084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:16.280099 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:16.280149 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:16.281134 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.281654 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.282731 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.283043 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.283133 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.285813 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:16.285816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:16.285816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:16.285825 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:16.287721 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:16.288246 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:16.288247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:16.288248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:16.288321 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:16.288685 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:16.288686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:16.288686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:16.288856 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:16.288938 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:16.289264 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 05:47:16.289363 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:47:16.289618 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.279182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:55027)
0926 05:47:16.279351 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.279407 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:16.279971 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:16.280218 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:16.280222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:16.280817 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:16.281031 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.281036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.281755 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.281756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.282635 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.282637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.283144 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.283145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.283278 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.284006 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:16.284016 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:16.285691 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:16.287828 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:16.287836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:16.287839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:16.288145 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:16.288476 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:16.288477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:16.288478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:16.288614 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:16.288960 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:16.288966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:16.289102 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 05:47:16.289198 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:47:16.289336 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260926 05:47:16.290477 20764 heartbeater.cc:347] Connected to a master server at 127.11.247.62:41141
I20260926 05:47:16.290684 20764 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:16.290902 20764 heartbeater.cc:511] Master 127.11.247.62:41141 requested a full tablet report, sending...
I20260926 05:47:16.291326 20439 ts_manager.cc:194] Registered new tserver with Master: 56e56ab1b1134e918acd5dbce26e9873 (127.11.247.2:34111)
I20260926 05:47:16.291754 20439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:55027
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:47:16.313452 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41141
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:16.398468 20769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:16.398742 20769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:16.398768 20769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:16.400120 20769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:16.400171 20769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:16.400199 20769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:16.401664 20769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:41141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:16.401872 20769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:16.402071 20769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:16.407361 20775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.407332 20777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.407315 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.407613 20769 server_base.cc:1034] running on GCE node
I20260926 05:47:16.407848 20769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:16.408062 20769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:16.409245 20769 hybrid_clock.cc:648] HybridClock initialized: now 1790401636409236 us; error 36 us; skew 500 ppm
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:16.411885 20769 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:47:16.412385 20769 webserver.cc:460] Webserver started at http://127.11.247.3:33295/ using document root <none> and password file <none>
I20260926 05:47:16.412632 20769 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:16.412680 20769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:16.412833 20769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:16.413910 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "badb86d93d1047b0a0b4aca29dcd8ce5"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "4b0ca5a7a7bd315a832dfd06e3d60b1f"
server_key_iv: "4f3e23f526c9a296fca3c6d1d3590365"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.414288 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "badb86d93d1047b0a0b4aca29dcd8ce5"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "4b0ca5a7a7bd315a832dfd06e3d60b1f"
server_key_iv: "4f3e23f526c9a296fca3c6d1d3590365"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.415783 20769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:47:16.416676 20784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.416911 20769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.417008 20769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "badb86d93d1047b0a0b4aca29dcd8ce5"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "4b0ca5a7a7bd315a832dfd06e3d60b1f"
server_key_iv: "4f3e23f526c9a296fca3c6d1d3590365"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.417101 20769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:16.431628 20769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:16.433923 20769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:16.434104 20769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:16.434391 20769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:16.434796 20769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:16.434868 20769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.434907 20769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:16.434926 20769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.441067 20769 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:37799
I20260926 05:47:16.441104 20897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:37799 every 8 connection(s)
I20260926 05:47:16.441478 20769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:16.448793 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20769
I20260926 05:47:16.448918 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:16.449126 12252 external_mini_cluster.cc:1428] Setting key 61268f8d8d971b70a907d72cc9fc2135
I20260926 05:47:16.451785 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.442255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:41025 (local address 127.11.247.62:41141)
0926 05:47:16.442327 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.442329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.442564 (+ 235us) server_negotiation.cc:407] Connection header received
0926 05:47:16.443105 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:16.443106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:16.443118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:16.443182 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:16.444147 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.444659 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.445722 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.445913 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.445970 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.448382 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:16.448383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:16.448383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:16.448391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:16.450199 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:16.450738 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:16.450739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:16.450739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:16.450794 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:16.451107 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:16.451108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:16.451108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:16.451216 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:16.451264 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:16.451652 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 05:47:16.451717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:47:16.452020 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.442244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:41025)
0926 05:47:16.442425 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.442473 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:16.443013 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:16.443234 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:16.443238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:16.443819 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:16.444036 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.444041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.444793 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.444795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.445632 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.445634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.445996 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.445997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.446102 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.446768 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:16.446777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:16.448254 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:16.450304 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:16.450310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:16.450312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:16.450559 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:16.450910 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:16.450911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:16.450911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:16.451033 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:16.451275 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:16.451280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:16.451467 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 05:47:16.451616 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 05:47:16.451762 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260926 05:47:16.452576 20898 heartbeater.cc:347] Connected to a master server at 127.11.247.62:41141
I20260926 05:47:16.452696 20898 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:16.452906 20898 heartbeater.cc:511] Master 127.11.247.62:41141 requested a full tablet report, sending...
I20260926 05:47:16.453394 20439 ts_manager.cc:194] Registered new tserver with Master: badb86d93d1047b0a0b4aca29dcd8ce5 (127.11.247.3:37799)
I20260926 05:47:16.453816 20439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:41025
I20260926 05:47:16.461267 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab.
Entry for principal oryx/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab.
WARNING: no policy specified for oryx/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab.
Entry for principal oryx/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab.
WARNING: no policy specified for oryx/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab.
Entry for principal oryx/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab.
I20260926 05:47:16.535655 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20501
I20260926 05:47:16.540573 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20635
I20260926 05:47:16.545543 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20769
I20260926 05:47:16.549782 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20407
I20260926 05:47:16.555172 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:41141
--webserver_interface=127.11.247.62
--webserver_port=35201
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:41141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:16.638725 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:16.638965 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:16.638990 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:16.640228 20907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:16.640277 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:16.640291 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:16.640304 20907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:16.640316 20907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:16.641904 20907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:41141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=35201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:16.642194 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:16.642457 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:16.647718 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.647819 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.647929 20907 server_base.cc:1034] running on GCE node
W20260926 05:47:16.647693 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.648286 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:16.648628 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:16.649807 20907 hybrid_clock.cc:648] HybridClock initialized: now 1790401636649769 us; error 55 us; skew 500 ppm
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:16.652500 20907 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:47:16.653033 20907 webserver.cc:460] Webserver started at http://127.11.247.62:35201/ using document root <none> and password file <none>
I20260926 05:47:16.653286 20907 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:16.653374 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:16.654974 20907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.655838 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.656112 20907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.656231 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "bf7fb445098a4ed3a9cfb939df0cca48"
format_stamp: "Formatted at 2026-09-26 05:47:15 on dist-test-slave-21p4"
server_key: "8eabdd31f3b8685f1f9963fddfd40b99"
server_key_iv: "ff7b439cd53ed0216093121cf32b7beb"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.656610 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:16.675374 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:16.677966 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:16.678148 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:16.682571 20907 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:41141
I20260926 05:47:16.682643 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:41141 every 8 connection(s)
I20260926 05:47:16.683117 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:16.684674 20975 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:47:16.685313 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Bootstrap starting.
I20260926 05:47:16.690491 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20907
I20260926 05:47:16.690896 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:45383
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:41141
--webserver_port=37859
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:47:16.694165 20975 log.cc:824] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:16.696724 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:47:16.697095 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Bootstrap complete.
I20260926 05:47:16.701399 20975 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:16.701860 20975 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fb445098a4ed3a9cfb939df0cca48, State: Initialized, Role: FOLLOWER
I20260926 05:47:16.702073 20975 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:16.702171 20975 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:16.702230 20975 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:16.702292 20975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:47:16.703822 20975 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:16.703944 20975 leader_election.cc:302] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7fb445098a4ed3a9cfb939df0cca48; no voters:
I20260926 05:47:16.704131 20975 leader_election.cc:288] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:47:16.704205 20979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:47:16.704386 20979 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [term 2 LEADER]: Becoming Leader. State: Replica: bf7fb445098a4ed3a9cfb939df0cca48, State: Running, Role: LEADER
I20260926 05:47:16.704391 20975 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:16.704566 20979 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } }
I20260926 05:47:16.704880 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } } }
I20260926 05:47:16.704983 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:16.705120 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7fb445098a4ed3a9cfb939df0cca48. Latest consensus state: current_term: 2 leader_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7fb445098a4ed3a9cfb939df0cca48" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 41141 } } }
I20260926 05:47:16.705260 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:16.706689 20995 catalog_manager.cc:1259] Loaded cluster ID: a485b2563744463e89d398c0a72c8e34
I20260926 05:47:16.706749 20995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: loading cluster ID for follower catalog manager: success
I20260926 05:47:16.707612 20995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: acquiring CA information for follower catalog manager: success
I20260926 05:47:16.707988 20995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:47:16.708122 20984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:16.708314 20984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:16.708415 20984 catalog_manager.cc:1259] Loaded cluster ID: a485b2563744463e89d398c0a72c8e34
I20260926 05:47:16.708448 20984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:16.708624 20984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:16.708737 20984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf7fb445098a4ed3a9cfb939df0cca48: Loaded TSK: 0
I20260926 05:47:16.708877 20984 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:47:16.792660 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:16.792928 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:16.792972 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:16.794201 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:16.794273 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:16.794322 20977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:16.795938 20977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.1.keytab
--principal=oryx/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:45383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=37859
--tserver_master_addrs=127.11.247.62:41141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:16.796221 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:16.796502 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:16.802758 21001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.802781 21002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.802875 21004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.803490 20977 server_base.cc:1034] running on GCE node
I20260926 05:47:16.803757 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:16.804044 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:16.805205 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790401636805178 us; error 31 us; skew 500 ppm
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:16.808302 20977 init.cc:377] Logged in from keytab as oryx/127.11.247.1@KRBTEST.COM (short username oryx)
I20260926 05:47:16.808877 20977 webserver.cc:460] Webserver started at http://127.11.247.1:37859/ using document root <none> and password file <none>
I20260926 05:47:16.809136 20977 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:16.809199 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:16.810817 20977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:16.811520 21011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.811681 20977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.811791 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "d91395b812844b4ca3d625f964d49319"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "dc9681220ff722e1b52f2a0ca65d7999"
server_key_iv: "cd5d1d217c7a29d5caeeb6aa2eb00986"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.812175 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:16.833325 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:16.836210 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:16.836619 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:16.837280 20977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:16.837990 20977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:16.838037 20977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.838081 20977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:16.838094 20977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.846027 20977 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:45383
I20260926 05:47:16.846057 21124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:45383 every 8 connection(s)
I20260926 05:47:16.846484 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:16.847345 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 20977
I20260926 05:47:16.847656 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:34111
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:41141
--webserver_port=32827
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:16.864427 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.847327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:60059)
0926 05:47:16.848430 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.848485 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:16.850360 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:16.851522 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:16.851528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:16.852490 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:16.852855 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.852862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.854556 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.854557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.855926 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.855930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.860004 (+ 4074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.860006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.860168 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.861127 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:16.861143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:16.864182 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":88,"threads_started":1}
W20260926 05:47:16.864670 21125 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:41141 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.247.62:41141: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
W20260926 05:47:16.864831 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:16.847705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:60059 (local address 127.11.247.62:41141)
0926 05:47:16.847923 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.847933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.848741 (+ 808us) server_negotiation.cc:407] Connection header received
0926 05:47:16.851179 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:16.851189 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:16.851268 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:16.851443 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:16.853001 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.854447 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.858822 (+ 4375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.859879 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.860022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.864330 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:60059: BlockingRecv error: recv got EOF from 127.11.247.1:60059 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:16.871210 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.864913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:33047)
0926 05:47:16.865148 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.865158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:16.865212 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:16.865349 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:16.865349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:16.865489 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:16.865546 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.865548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.866554 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.866556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.867612 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.867616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.868189 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.868191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.868283 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.868789 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:16.868801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:16.871136 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216}
W20260926 05:47:16.871694 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:16.865004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:33047 (local address 127.11.247.62:41141)
0926 05:47:16.865068 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.865070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.865194 (+ 124us) server_negotiation.cc:407] Connection header received
0926 05:47:16.865276 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:16.865277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:16.865291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:16.865337 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:16.865625 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.866417 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.867710 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.868096 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.868182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.871435 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:33047: BlockingRecv error: recv got EOF from 127.11.247.1:33047 (error 108)
Metrics: {"server-negotiator.queue_time_us":45,"thread_start_us":106,"threads_started":1}
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:16.876616 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.871478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:59145)
0926 05:47:16.871619 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.871633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:16.871675 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:16.871907 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:16.871908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:16.872072 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:16.872116 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.872118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.872981 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.872982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.873564 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:16.873566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.873947 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.873947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:16.874005 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.874456 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:16.874461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:16.876521 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260926 05:47:16.876931 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:16.871523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:59145 (local address 127.11.247.62:41141)
0926 05:47:16.871810 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.871812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.871819 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:47:16.871841 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:16.871842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:16.871853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:16.871899 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:16.872200 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.872906 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.873652 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:16.873862 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:16.873943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:16.876853 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:59145: BlockingRecv error: recv got EOF from 127.11.247.1:59145 (error 108)
Metrics: {"server-negotiator.queue_time_us":269}
W20260926 05:47:16.878202 21125 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:47:16.950933 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:16.951269 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:16.951313 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:16.953819 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:16.953915 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:16.953966 21129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:16.955924 21129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.2.keytab
--principal=oryx/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:34111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=32827
--tserver_master_addrs=127.11.247.62:41141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:16.956287 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:16.956553 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:16.963860 21136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.963824 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:16.964011 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:16.964491 21129 server_base.cc:1034] running on GCE node
I20260926 05:47:16.964732 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:16.965010 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:16.966207 21129 hybrid_clock.cc:648] HybridClock initialized: now 1790401636966189 us; error 32 us; skew 500 ppm
Sep 26 05:47:16 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401636, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:16.969485 21129 init.cc:377] Logged in from keytab as oryx/127.11.247.2@KRBTEST.COM (short username oryx)
I20260926 05:47:16.970136 21129 webserver.cc:460] Webserver started at http://127.11.247.2:32827/ using document root <none> and password file <none>
I20260926 05:47:16.970440 21129 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:16.970541 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:16.972262 21129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.973045 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.973255 21129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:16.973377 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "56e56ab1b1134e918acd5dbce26e9873"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "6b5ea9e773507502fa41226d60fb2c00"
server_key_iv: "212bfa1411063fdbcfd609c4d1bcc402"
server_key_version: "encryptionkey@0"
I20260926 05:47:16.973855 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:16.986006 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:16.988544 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:16.988739 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:16.989012 21129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:16.989457 21129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:16.989522 21129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.989571 21129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:16.989603 21129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:16.997963 21129 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:34111
I20260926 05:47:16.997998 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:34111 every 8 connection(s)
I20260926 05:47:16.998478 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.007230 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21129
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.007627 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:37799
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:41141
--webserver_port=33295
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:17.008806 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:16.999470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:49045 (local address 127.11.247.62:41141)
0926 05:47:16.999553 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:16.999556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:16.999795 (+ 239us) server_negotiation.cc:407] Connection header received
0926 05:47:17.000319 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.000320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.000339 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.000451 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.001634 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.002609 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.003686 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.004045 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.004128 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.008714 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:49045: BlockingRecv error: recv got EOF from 127.11.247.2:49045 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:47:17.008868 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:16.999459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:49045)
0926 05:47:16.999651 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:16.999697 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.000224 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.000504 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.000507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.001256 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.001522 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.001528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.002728 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.002729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.003592 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.003594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.004256 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.004257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.004418 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.005158 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.005168 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.008581 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
W20260926 05:47:17.009498 21260 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:41141 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.247.62:41141: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.015434 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.009775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:49051)
0926 05:47:17.009840 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:17.009851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.009896 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.010108 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.010110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.010315 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.010394 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.010396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.011262 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.011263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.012197 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.012200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.012578 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.012579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.012669 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.013146 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.013153 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.015335 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 05:47:17.015493 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:17.009838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:49051 (local address 127.11.247.62:41141)
0926 05:47:17.009886 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.009888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.009899 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:17.009970 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.009971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.009983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.010051 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.010543 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.011162 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.012294 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.012492 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.012555 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.015432 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:49051: BlockingRecv error: recv got EOF from 127.11.247.2:49051 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.020380 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.015927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:39405)
0926 05:47:17.015972 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:17.015983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.016023 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.016163 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.016164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.016312 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.016344 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.016345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.017020 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.017020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.017534 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.017535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.017898 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.017899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.017953 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.018534 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.018538 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.020301 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 05:47:17.020659 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:17.015997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:39405 (local address 127.11.247.62:41141)
0926 05:47:17.016042 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.016044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.016052 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:17.016077 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.016077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.016087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.016158 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.016419 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.016914 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.017618 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.017814 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.017904 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.020610 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:39405: BlockingRecv error: recv got EOF from 127.11.247.2:39405 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:47:17.020661 21260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:47:17.098810 21263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:17.099082 21263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:17.099123 21263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:17.100353 21263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:17.100432 21263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:17.100482 21263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:17.102146 21263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/krb5kdc/oryx_127.11.247.3.keytab
--principal=oryx/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:37799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=33295
--tserver_master_addrs=127.11.247.62:41141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:17.102430 21263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:17.102732 21263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:17.108458 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:17.108464 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:17.108453 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:17.108698 21263 server_base.cc:1034] running on GCE node
I20260926 05:47:17.108999 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:17.109220 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:17.110397 21263 hybrid_clock.cc:648] HybridClock initialized: now 1790401637110369 us; error 44 us; skew 500 ppm
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401637, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:17.113422 21263 init.cc:377] Logged in from keytab as oryx/127.11.247.3@KRBTEST.COM (short username oryx)
I20260926 05:47:17.114076 21263 webserver.cc:460] Webserver started at http://127.11.247.3:33295/ using document root <none> and password file <none>
I20260926 05:47:17.114352 21263 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:17.114414 21263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:17.116040 21263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:17.116779 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:17.117071 21263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:17.117146 21263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "badb86d93d1047b0a0b4aca29dcd8ce5"
format_stamp: "Formatted at 2026-09-26 05:47:16 on dist-test-slave-21p4"
server_key: "4b0ca5a7a7bd315a832dfd06e3d60b1f"
server_key_iv: "4f3e23f526c9a296fca3c6d1d3590365"
server_key_version: "encryptionkey@0"
I20260926 05:47:17.117580 21263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:17.140081 21263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:17.142648 21263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:17.142843 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:17.143151 21263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:17.143501 21263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:17.143536 21263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:17.143560 21263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:17.143573 21263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:17.149683 21263 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:37799
I20260926 05:47:17.149742 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:37799 every 8 connection(s)
I20260926 05:47:17.150175 21263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:17.154336 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21263
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:47:17.158406 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:17.151110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:33755 (local address 127.11.247.62:41141)
0926 05:47:17.151187 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.151189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.151457 (+ 268us) server_negotiation.cc:407] Connection header received
0926 05:47:17.151873 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.151874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.151885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.151956 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.152929 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.153424 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.154817 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.155123 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.155213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.158339 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:33755: BlockingRecv error: recv got EOF from 127.11.247.3:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:47:17.158468 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.151032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:33755)
0926 05:47:17.151213 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:17.151256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.151759 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.152014 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.152017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.152611 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.152831 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.152836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.153531 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.153532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.154704 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.154707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.155226 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.155227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.155366 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.155937 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.155946 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.158209 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260926 05:47:17.158695 21392 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:41141 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.247.62:41141: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.162799 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.154982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.11.247.62:41141)
0926 05:47:17.155048 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.155050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.155100 (+ 50us) server_negotiation.cc:407] Connection header received
0926 05:47:17.155174 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.155174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.155187 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.155251 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.155987 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.156488 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.157366 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.157633 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.157707 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.159229 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:17.159233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:17.159233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:17.159245 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:17.161073 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:17.161508 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:17.161511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:17.161512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:17.161648 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:17.161933 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:17.161934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:17.161934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:17.162193 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:17.162300 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:17.162528 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 05:47:17.162685 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.164083 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.158981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:49175)
0926 05:47:17.159026 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:17.159037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.159073 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.159518 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.159519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.159695 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.159741 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.159743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.160476 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.160477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.161058 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.161060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.161444 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.161445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.161518 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.161895 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.161900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.164001 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:47:17.164186 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:17.159008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:49175 (local address 127.11.247.62:41141)
0926 05:47:17.159047 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.159050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.159079 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:47:17.159127 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.159128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.159139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.159197 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.159829 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.160399 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.161153 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.161342 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.161430 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.164112 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:49175: BlockingRecv error: recv got EOF from 127.11.247.3:49175 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.168573 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.164402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:41127)
0926 05:47:17.164459 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:17.164469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.164506 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.164625 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.164626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.164757 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.164786 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.164787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.165430 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.165431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.165932 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.165933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.166322 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.166323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.166384 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.166866 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.166871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.168489 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 05:47:17.168617 21128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:17.164485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:41127 (local address 127.11.247.62:41141)
0926 05:47:17.164536 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.164538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.164546 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:47:17.164566 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.164566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.164575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.164623 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.164884 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.165347 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.166038 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.166237 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.166321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.168557 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:41127: BlockingRecv error: recv got EOF from 127.11.247.3:41127 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 05:47:17.168797 21392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:17 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:17.883625 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:17.878798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:58593)
0926 05:47:17.878952 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:17.878966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:17.879042 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:17.879206 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:17.879207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:17.879395 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:17.879448 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.879450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.880120 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.880121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.880761 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:17.880763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.881222 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.881223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:17.881268 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.881719 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:17.881723 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:17.883550 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
W20260926 05:47:17.883791 21397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:17.878885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:58593 (local address 127.11.247.62:41141)
0926 05:47:17.879037 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:17.879039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:17.879050 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:17.879109 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:17.879109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:17.879121 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:17.879187 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:17.879536 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.880042 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.880896 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:17.881160 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:17.881213 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:17.883680 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:58593: BlockingRecv error: recv got EOF from 127.11.247.1:58593 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
Sep 26 05:47:18 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:18 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:18.026005 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:18.021078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:53935)
0926 05:47:18.021216 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:18.021229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:18.021295 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:18.021471 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:18.021472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:18.021659 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:18.021708 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:18.021709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.022383 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.022385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:18.022976 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:18.022978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.023349 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.023350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:18.023421 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:18.023971 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:18.023980 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:18.025930 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
W20260926 05:47:18.026160 21397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:18.021189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:53935 (local address 127.11.247.62:41141)
0926 05:47:18.021264 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:18.021266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:18.021279 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:47:18.021386 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:18.021387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:18.021397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:18.021463 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:18.021792 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.022296 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.023075 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.023272 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.023351 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:18.026056 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:53935: BlockingRecv error: recv got EOF from 127.11.247.2:53935 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 05:47:18 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:18 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:18.174391 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:18.169218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:34217)
0926 05:47:18.169398 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:18.169410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:18.169493 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:18.169676 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:18.169676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:18.169881 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:18.169935 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:18.169937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.170741 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.170743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:18.171325 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:18.171327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.171802 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.171804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:18.171869 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:18.172379 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:18.172384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:18.174276 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
W20260926 05:47:18.174431 21397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:18.169295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:34217 (local address 127.11.247.62:41141)
0926 05:47:18.169366 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:18.169369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:18.169469 (+ 100us) server_negotiation.cc:407] Connection header received
0926 05:47:18.169563 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:18.169564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:18.169580 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:18.169665 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:18.170025 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.170640 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.171443 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.171673 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.171808 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:18.174365 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:34217: BlockingRecv error: recv got EOF from 127.11.247.3:34217 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 05:47:18 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:18 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:18.889243 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:18.884292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:47247)
0926 05:47:18.884481 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:18.884499 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:18.884554 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:18.884753 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:18.884754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:18.884929 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:18.884987 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:18.884988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.885689 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.885690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:18.886244 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:18.886246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.886816 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.886817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:18.886872 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:18.887338 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:18.887343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:18.889180 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260926 05:47:18.889452 21401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:18.884381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:47247 (local address 127.11.247.62:41141)
0926 05:47:18.884621 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:18.884623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:18.884633 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:18.884657 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:18.884657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:18.884668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:18.884735 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:18.885070 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.885616 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.886358 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:18.886675 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:18.886753 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:18.889310 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:47247: BlockingRecv error: recv got EOF from 127.11.247.1:47247 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":181,"threads_started":1}
Sep 26 05:47:19 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:19 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:19.031678 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:19.026542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:54303)
0926 05:47:19.026727 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:19.026741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:19.026840 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:19.026992 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:19.026992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:19.027188 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:19.027239 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:19.027241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.027945 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.027946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:19.028497 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:19.028499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.028884 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.028885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:19.028947 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:19.029491 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:19.029510 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:19.031580 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
W20260926 05:47:19.031750 21401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:19.026662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:54303 (local address 127.11.247.62:41141)
0926 05:47:19.026731 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:19.026734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:19.026820 (+ 86us) server_negotiation.cc:407] Connection header received
0926 05:47:19.026909 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:19.026910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:19.026924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:19.026983 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:19.027328 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.027854 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.028600 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.028807 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.028886 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:19.031686 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:54303: BlockingRecv error: recv got EOF from 127.11.247.2:54303 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 05:47:19 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:19 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:19.179665 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:19.174968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:52125)
0926 05:47:19.175126 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:19.175137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:19.175188 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:19.175336 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:19.175337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:19.175537 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:19.175588 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:19.175590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.176228 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.176228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:19.176761 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:19.176764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.177090 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.177091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:19.177132 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:19.177673 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:19.177678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:19.179591 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
W20260926 05:47:19.179781 21401 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:19.175119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:52125 (local address 127.11.247.62:41141)
0926 05:47:19.175188 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:19.175190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:19.175238 (+ 48us) server_negotiation.cc:407] Connection header received
0926 05:47:19.175260 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:19.175261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:19.175273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:19.175315 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:19.175654 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.176162 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.176836 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.177026 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.177079 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:19.179697 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:52125: BlockingRecv error: recv got EOF from 127.11.247.3:52125 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 05:47:19 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:19 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:19.895449 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:19.889824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:43847)
0926 05:47:19.890025 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:19.890037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:19.890097 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:19.890305 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:19.890307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:19.890738 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:19.890795 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:19.890798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.891568 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.891569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:19.892117 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:19.892119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.892550 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.892551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:19.892606 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:19.893080 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:19.893085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:19.895303 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
W20260926 05:47:19.895607 21405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:19.889954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:43847 (local address 127.11.247.62:41141)
0926 05:47:19.890097 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:19.890099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:19.890117 (+ 18us) server_negotiation.cc:407] Connection header received
0926 05:47:19.890160 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:19.890161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:19.890180 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:19.890241 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:19.890912 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.891476 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.892224 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:19.892464 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:19.892526 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:19.895508 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:43847: BlockingRecv error: recv got EOF from 127.11.247.1:43847 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
Sep 26 05:47:20 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:20 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:20.037318 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:20.032301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:55641)
0926 05:47:20.032483 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:20.032497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:20.032571 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:20.032767 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:20.032768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:20.032956 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:20.033011 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:20.033013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.033722 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.033723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:20.034282 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:20.034284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.034736 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.034737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:20.034809 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:20.035322 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:20.035326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:20.037236 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
W20260926 05:47:20.037454 21405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:20.032446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:55641 (local address 127.11.247.62:41141)
0926 05:47:20.032505 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:20.032507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:20.032551 (+ 44us) server_negotiation.cc:407] Connection header received
0926 05:47:20.032667 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:20.032668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:20.032680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:20.032742 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:20.033107 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.033629 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.034380 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.034649 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.034726 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:20.037385 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:55641: BlockingRecv error: recv got EOF from 127.11.247.2:55641 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 05:47:20 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:20 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:20.185927 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:20.180231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:50553)
0926 05:47:20.180423 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:20.180442 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:20.180502 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:20.180688 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:20.180689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:20.180899 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:20.180960 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:20.180962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.181693 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.181694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:20.182356 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:20.182359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.182811 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.182812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:20.182876 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:20.183563 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:20.183568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:20.185857 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":96,"threads_started":1}
W20260926 05:47:20.186281 21405 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:20.180364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:50553 (local address 127.11.247.62:41141)
0926 05:47:20.180432 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:20.180434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:20.180506 (+ 72us) server_negotiation.cc:407] Connection header received
0926 05:47:20.180549 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:20.180550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:20.180571 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:20.180630 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:20.181053 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.181565 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.182485 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.182715 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.182784 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:20.185997 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:50553: BlockingRecv error: recv got EOF from 127.11.247.3:50553 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 05:47:20 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:20 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:20.908123 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:20.896199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:44351)
0926 05:47:20.896384 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:20.896396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:20.896485 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:20.896821 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:20.896822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:20.897087 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:20.897157 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:20.897159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.897952 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.897953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:20.898674 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:20.898678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.899085 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.899086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:20.899140 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:20.905780 (+ 6640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:20.905788 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:20.908005 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
W20260926 05:47:20.908237 21409 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:20.896371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:44351 (local address 127.11.247.62:41141)
0926 05:47:20.896511 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:20.896513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:20.896526 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:47:20.896581 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:20.896581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:20.896595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:20.896781 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:20.897326 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.897862 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.898782 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:20.898990 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:20.899057 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:20.908154 (+ 9097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:44351: BlockingRecv error: recv got EOF from 127.11.247.1:44351 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
Sep 26 05:47:21 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:21 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:21.044512 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:21.038036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:38327)
0926 05:47:21.038207 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:21.038219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:21.038286 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:21.038499 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:21.038499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:21.038692 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:21.038747 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:21.038748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.039535 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.039536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:21.040110 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:21.040112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.040623 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.040624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:21.040687 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:21.041284 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:21.041290 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:21.044446 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
W20260926 05:47:21.044699 21409 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:21.038253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:38327 (local address 127.11.247.62:41141)
0926 05:47:21.038335 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:21.038338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:21.038348 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:21.038383 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:21.038384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:21.038396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:21.038470 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:21.038844 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.039443 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.040274 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.040504 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.040590 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:21.044593 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:38327: BlockingRecv error: recv got EOF from 127.11.247.2:38327 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 05:47:21 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:21 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:21.192124 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:21.186891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:46325)
0926 05:47:21.187162 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:21.187176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:21.187257 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:21.187458 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:21.187459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:21.187638 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:21.187713 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:21.187715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.188625 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.188626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:21.189151 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:21.189153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.189571 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.189572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:21.189626 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:21.190166 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:21.190171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:21.192062 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":81,"threads_started":1}
W20260926 05:47:21.192245 21409 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:21.187138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:46325 (local address 127.11.247.62:41141)
0926 05:47:21.187236 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:21.187239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:21.187254 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:47:21.187336 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:21.187338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:21.187354 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:21.187426 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:21.187825 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.188536 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.189252 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.189459 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.189535 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:21.192164 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:46325: BlockingRecv error: recv got EOF from 127.11.247.3:46325 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 05:47:21 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:21 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:21.914288 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:21.908779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:56977)
0926 05:47:21.908963 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:21.908973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:21.909027 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:21.909185 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:21.909185 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:21.909356 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:21.909409 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:21.909410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.910160 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.910161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:21.910786 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:21.910789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.911168 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.911169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:21.911218 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:21.911739 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:21.911744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:21.914185 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
W20260926 05:47:21.914374 21413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:21.908890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:56977 (local address 127.11.247.62:41141)
0926 05:47:21.909018 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:21.909020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:21.909031 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:21.909079 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:21.909079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:21.909091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:21.909163 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:21.909490 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.910040 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.910874 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:21.911086 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:21.911146 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:21.914299 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:56977: BlockingRecv error: recv got EOF from 127.11.247.1:56977 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
Sep 26 05:47:22 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:22 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:22.050554 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:22.045162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:35459)
0926 05:47:22.045322 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:22.045334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:22.045416 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:22.045685 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:22.045685 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:22.045874 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:22.045930 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:22.045932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.046690 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.046691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:22.047362 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:22.047366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.047813 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.047814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:22.047886 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:22.048358 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:22.048364 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:22.050491 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
W20260926 05:47:22.050750 21413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:22.045434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:35459 (local address 127.11.247.62:41141)
0926 05:47:22.045508 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:22.045510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:22.045557 (+ 47us) server_negotiation.cc:407] Connection header received
0926 05:47:22.045589 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:22.045590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:22.045600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:22.045660 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:22.046014 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.046558 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.047461 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.047698 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.047795 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:22.050686 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:35459: BlockingRecv error: recv got EOF from 127.11.247.2:35459 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 05:47:22 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:22 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:22.197758 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:22.192874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:58203)
0926 05:47:22.193025 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:22.193041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:22.193102 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:22.193310 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:22.193311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:22.193515 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:22.193565 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:22.193567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.194298 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.194299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:22.194878 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:22.194881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.195263 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.195264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:22.195335 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:22.195854 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:22.195859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:22.197654 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260926 05:47:22.197819 21413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:22.192967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:58203 (local address 127.11.247.62:41141)
0926 05:47:22.193053 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:22.193055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:22.193089 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:47:22.193166 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:22.193166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:22.193180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:22.193247 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:22.193661 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.194204 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.194976 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.195179 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.195262 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:22.197754 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:58203: BlockingRecv error: recv got EOF from 127.11.247.3:58203 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 05:47:22 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:22 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:22.920665 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:22.915107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:51535)
0926 05:47:22.915330 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:22.915342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:22.915413 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:22.915595 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:22.915596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:22.915793 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:22.915845 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:22.915847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.916827 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.916828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:22.917416 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:22.917418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.917808 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.917809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:22.917863 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:22.918413 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:22.918418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:22.920548 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
W20260926 05:47:22.920784 21417 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:22.915221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:51535 (local address 127.11.247.62:41141)
0926 05:47:22.915366 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:22.915368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:22.915397 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:47:22.915463 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:22.915463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:22.915475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:22.915565 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:22.915927 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.916729 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.917499 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:22.917698 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:22.917763 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:22.920669 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:51535: BlockingRecv error: recv got EOF from 127.11.247.1:51535 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
Sep 26 05:47:23 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:23 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:23.056595 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:23.051181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:42541)
0926 05:47:23.051402 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:23.051416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:23.051488 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:23.051677 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:23.051678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:23.051837 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:23.051904 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:23.051906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.052624 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.052625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:23.053209 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:23.053211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.053677 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.053678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:23.053732 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:23.054302 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:23.054307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:23.056531 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
W20260926 05:47:23.056792 21417 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:23.051320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:42541 (local address 127.11.247.62:41141)
0926 05:47:23.051409 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:23.051411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:23.051483 (+ 72us) server_negotiation.cc:407] Connection header received
0926 05:47:23.051556 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:23.051557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:23.051569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:23.051630 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:23.051981 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.052529 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.053381 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.053582 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.053662 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:23.056719 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:42541: BlockingRecv error: recv got EOF from 127.11.247.2:42541 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 05:47:23 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:23 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:23.203258 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:23.198403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:37371)
0926 05:47:23.198679 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:23.198699 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:23.198778 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:23.198923 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:23.198924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:23.199090 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:23.199143 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:23.199145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.199799 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.199800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:23.200347 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:23.200349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.200691 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.200692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:23.200736 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:23.201294 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:23.201299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:23.203191 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
W20260926 05:47:23.203459 21417 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:23.198491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:37371 (local address 127.11.247.62:41141)
0926 05:47:23.198703 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:23.198705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:23.198760 (+ 55us) server_negotiation.cc:407] Connection header received
0926 05:47:23.198842 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:23.198843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:23.198856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:23.198897 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:23.199213 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.199715 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.200438 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.200624 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.200677 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:23.203388 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:37371: BlockingRecv error: recv got EOF from 127.11.247.3:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Sep 26 05:47:23 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:23 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:23.927047 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:23.921186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:60645)
0926 05:47:23.921512 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:23.921527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:23.921619 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:23.922021 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:23.922022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:23.922206 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:23.922266 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:23.922267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.923111 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.923112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:23.923836 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:23.923839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.924194 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.924194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:23.924240 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:23.924815 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:23.924821 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:23.926927 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":244,"threads_started":1}
W20260926 05:47:23.927271 21421 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:23.921466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:60645 (local address 127.11.247.62:41141)
0926 05:47:23.921612 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:23.921614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:23.921626 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:47:23.921685 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:23.921686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:23.921703 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:23.921996 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:23.922395 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.923022 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.923920 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:23.924119 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:23.924177 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:23.927167 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:60645: BlockingRecv error: recv got EOF from 127.11.247.1:60645 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
Sep 26 05:47:24 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:24 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:24.062893 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:24.057239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:47267)
0926 05:47:24.057462 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:24.057484 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:24.057565 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:24.057764 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:24.057765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:24.057938 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:24.058004 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:24.058005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.058792 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.058793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:24.059333 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:24.059335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.059885 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.059886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:24.059953 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:24.060444 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:24.060449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:24.062828 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20260926 05:47:24.063007 21421 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:24.057333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:47267 (local address 127.11.247.62:41141)
0926 05:47:24.057453 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:24.057455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:24.057602 (+ 147us) server_negotiation.cc:407] Connection header received
0926 05:47:24.057650 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:24.057651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:24.057664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:24.057710 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:24.058090 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.058658 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.059440 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.059793 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.059858 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:24.062935 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:47267: BlockingRecv error: recv got EOF from 127.11.247.2:47267 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 26 05:47:24 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:24 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:24.208845 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:24.203901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:56287)
0926 05:47:24.204051 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:24.204064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:24.204142 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:24.204277 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:24.204278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:24.204441 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:24.204494 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:24.204495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.205315 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.205316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:24.205846 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:24.205848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.206392 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.206393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:24.206453 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:24.207003 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:24.207009 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:24.208777 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
W20260926 05:47:24.209002 21421 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:24.203979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:56287 (local address 127.11.247.62:41141)
0926 05:47:24.204064 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:24.204066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:24.204141 (+ 75us) server_negotiation.cc:407] Connection header received
0926 05:47:24.204189 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:24.204190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:24.204204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:24.204257 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:24.204556 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.205248 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.205939 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.206250 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.206319 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:24.208899 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:56287: BlockingRecv error: recv got EOF from 127.11.247.3:56287 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 05:47:24 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:24 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:24.932521 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:24.927571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:46563)
0926 05:47:24.927761 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:24.927775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:24.927861 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:24.928059 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:24.928060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:24.928227 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:24.928290 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:24.928291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.928960 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.928961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:24.929505 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:24.929507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.929886 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.929886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:24.929926 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:24.930488 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:24.930492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:24.932410 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
W20260926 05:47:24.932580 21425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:24.927680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:46563 (local address 127.11.247.62:41141)
0926 05:47:24.927792 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:24.927794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:24.927842 (+ 48us) server_negotiation.cc:407] Connection header received
0926 05:47:24.927957 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:24.927957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:24.927970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:24.928034 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:24.928361 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.928886 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.929590 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:24.929804 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:24.929871 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:24.932502 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:46563: BlockingRecv error: recv got EOF from 127.11.247.1:46563 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
Sep 26 05:47:25 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:25 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:25.068475 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:25.063438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:47915)
0926 05:47:25.063635 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:25.063653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:25.063738 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:25.063900 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:25.063901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:25.064081 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:25.064130 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:25.064132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.065010 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.065012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:25.065542 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:25.065543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.065914 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.065915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:25.065996 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:25.066489 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:25.066493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:25.068350 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":121,"threads_started":1}
W20260926 05:47:25.068511 21425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:25.063521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:47915 (local address 127.11.247.62:41141)
0926 05:47:25.063610 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:25.063612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:25.063717 (+ 105us) server_negotiation.cc:407] Connection header received
0926 05:47:25.063817 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:25.063818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:25.063833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:25.063884 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:25.064221 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.064917 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.065628 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.065838 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.065916 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:25.068444 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:47915: BlockingRecv error: recv got EOF from 127.11.247.2:47915 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 05:47:25 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:25 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:25.214493 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:25.209435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:57365)
0926 05:47:25.209582 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:25.209597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:25.209659 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:25.209804 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:25.209805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:25.209979 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:25.210030 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:25.210032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.210799 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.210801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:25.211384 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:25.211387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.211783 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.211785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:25.211841 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:25.212375 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:25.212379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:25.214418 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260926 05:47:25.214684 21425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:25.209515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:57365 (local address 127.11.247.62:41141)
0926 05:47:25.209603 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:25.209605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:25.209658 (+ 53us) server_negotiation.cc:407] Connection header received
0926 05:47:25.209704 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:25.209705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:25.209718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:25.209784 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:25.210133 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.210695 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.211488 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.211681 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.211749 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:25.214557 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:57365: BlockingRecv error: recv got EOF from 127.11.247.3:57365 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 05:47:25 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:25 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:25.941857 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:25.933032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:34721)
0926 05:47:25.933207 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:25.933218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:25.933323 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:25.933510 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:25.933510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:25.933684 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:25.933736 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:25.933737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.934549 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.934550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:25.935126 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:25.935128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.935501 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.935502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:25.935547 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:25.939795 (+ 4248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:25.939802 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:25.941790 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
W20260926 05:47:25.942020 21429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:25.933126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:34721 (local address 127.11.247.62:41141)
0926 05:47:25.933237 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:25.933239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:25.933298 (+ 59us) server_negotiation.cc:407] Connection header received
0926 05:47:25.933411 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:25.933412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:25.933425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:25.933491 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:25.933831 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.934461 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.935220 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:25.935425 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:25.935489 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:25.941914 (+ 6425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:34721: BlockingRecv error: recv got EOF from 127.11.247.1:34721 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":57,"threads_started":1}
Sep 26 05:47:26 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:26 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:26.074074 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:26.069033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:52439)
0926 05:47:26.069180 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:26.069192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:26.069337 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:26.069471 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:26.069471 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:26.069634 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:26.069688 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:26.069689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.070388 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.070388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:26.070949 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:26.070950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.071355 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.071357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:26.071411 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:26.071961 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:26.071966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:26.074010 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260926 05:47:26.074350 21429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:26.069090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:52439 (local address 127.11.247.62:41141)
0926 05:47:26.069144 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:26.069146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:26.069316 (+ 170us) server_negotiation.cc:407] Connection header received
0926 05:47:26.069392 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:26.069392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:26.069406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:26.069451 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:26.069752 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.070265 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.071036 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.071232 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.071339 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:26.074274 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:52439: BlockingRecv error: recv got EOF from 127.11.247.2:52439 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 05:47:26 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:26 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:26.220790 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:26.215121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:38317)
0926 05:47:26.215325 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:26.215337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:26.215434 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:26.215697 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:26.215699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:26.215890 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:26.215941 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:26.215942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.216828 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.216829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:26.217373 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:26.217374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.217749 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.217750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:26.217801 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:26.218343 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:26.218348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:26.220702 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
W20260926 05:47:26.220985 21429 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:26.215221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:38317 (local address 127.11.247.62:41141)
0926 05:47:26.215268 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:26.215270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:26.215412 (+ 142us) server_negotiation.cc:407] Connection header received
0926 05:47:26.215499 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:26.215500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:26.215517 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:26.215650 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:26.216034 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.216710 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.217464 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.217659 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.217720 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:26.220902 (+ 3182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:38317: BlockingRecv error: recv got EOF from 127.11.247.3:38317 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 05:47:26 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:26 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:26.947973 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:26.942495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:58273)
0926 05:47:26.942712 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:26.942724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:26.942788 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:26.942963 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:26.942964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:26.943145 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:26.943200 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:26.943202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.943895 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.943896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:26.944487 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:26.944489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.944904 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.944905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:26.944967 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:26.945583 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:26.945587 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:26.947852 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":141,"threads_started":1}
W20260926 05:47:26.948019 21433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:26.942671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:58273 (local address 127.11.247.62:41141)
0926 05:47:26.942799 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:26.942801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:26.942818 (+ 17us) server_negotiation.cc:407] Connection header received
0926 05:47:26.942844 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:26.942845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:26.942856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:26.942919 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:26.943284 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.943808 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.944581 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:26.944801 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:26.944868 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:26.947944 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:58273: BlockingRecv error: recv got EOF from 127.11.247.1:58273 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
Sep 26 05:47:27 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:27 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:27.080569 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:27.074768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:51527)
0926 05:47:27.074903 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:27.074924 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:27.075004 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:27.075237 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:27.075238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:27.075468 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:27.075525 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:27.075527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.076241 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.076242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:27.076861 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:27.076864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.077408 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.077409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:27.077463 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:27.078111 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:27.078117 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:27.080480 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260926 05:47:27.080746 21433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:27.074860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:51527 (local address 127.11.247.62:41141)
0926 05:47:27.074912 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:27.074914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:27.075029 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:47:27.075065 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:27.075065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:27.075078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:27.075192 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:27.075628 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.076147 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.077024 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.077329 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.077399 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:27.080647 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:51527: BlockingRecv error: recv got EOF from 127.11.247.2:51527 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 05:47:27 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:27 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:27.226761 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:27.221387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:39961)
0926 05:47:27.221578 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:27.221591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:27.221808 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:27.222008 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:27.222010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:27.222232 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:27.222285 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:27.222287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.223225 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.223226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:27.223777 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:27.223779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.224161 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.224162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:27.224208 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:27.224702 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:27.224706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:27.226654 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
W20260926 05:47:27.226819 21433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:27.221578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:39961 (local address 127.11.247.62:41141)
0926 05:47:27.221637 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:27.221639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:27.221822 (+ 183us) server_negotiation.cc:407] Connection header received
0926 05:47:27.221858 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:27.221858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:27.221880 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:27.221944 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:27.222382 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.223120 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.223873 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.224067 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.224129 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:27.226743 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:39961: BlockingRecv error: recv got EOF from 127.11.247.3:39961 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 05:47:27 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:27 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:27.953974 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:27.948667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:37891)
0926 05:47:27.948829 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:27.948848 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:27.948903 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:27.949208 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:27.949209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:27.949454 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:27.949508 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:27.949510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.950426 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.950428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:27.951009 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:27.951010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.951388 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.951389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:27.951437 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:27.951898 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:27.951903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:27.953871 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
W20260926 05:47:27.954133 21437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:27.948839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:37891 (local address 127.11.247.62:41141)
0926 05:47:27.948981 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:27.948983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:27.948993 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:27.949021 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:27.949021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:27.949032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:27.949184 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:27.949598 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.950344 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.951089 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:27.951308 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:27.951368 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:27.953998 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:37891: BlockingRecv error: recv got EOF from 127.11.247.1:37891 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Sep 26 05:47:28 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:28 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:28.086861 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:28.081225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:55369)
0926 05:47:28.081496 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:28.081508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:28.081616 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:28.081828 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:28.081829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:28.082029 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:28.082078 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:28.082080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.082972 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.082973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:28.083601 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:28.083604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.084012 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.084013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:28.084078 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:28.084598 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:28.084603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:28.086752 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":142,"threads_started":1}
W20260926 05:47:28.087016 21437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:28.081373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:55369 (local address 127.11.247.62:41141)
0926 05:47:28.081483 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:28.081486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:28.081599 (+ 113us) server_negotiation.cc:407] Connection header received
0926 05:47:28.081729 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:28.081730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:28.081751 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:28.081819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:28.082209 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.082906 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.083703 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.083910 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.084008 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:28.086903 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:55369: BlockingRecv error: recv got EOF from 127.11.247.2:55369 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 05:47:28 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:28 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:28.232903 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:28.227377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:45037)
0926 05:47:28.227565 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:28.227595 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:28.227678 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:28.227889 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:28.227891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:28.228110 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:28.228160 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:28.228163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.228881 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.228883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:28.229483 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:28.229485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.229924 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.229925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:28.229995 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:28.230511 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:28.230516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:28.232839 (+ 2323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
W20260926 05:47:28.233086 21437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:28.227560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:45037 (local address 127.11.247.62:41141)
0926 05:47:28.227638 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:28.227640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:28.227656 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:47:28.227792 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:28.227793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:28.227810 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:28.227881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:28.228258 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.228785 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.229587 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.229803 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.229911 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:28.232979 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:45037: BlockingRecv error: recv got EOF from 127.11.247.3:45037 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 05:47:28 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:28 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:28.959928 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:28.954547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:51729)
0926 05:47:28.954771 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:28.954784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:28.954846 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:28.955026 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:28.955027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:28.955268 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:28.955319 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:28.955322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.956107 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.956107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:28.956699 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:28.956701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.957091 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.957092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:28.957142 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:28.957717 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:28.957722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:28.959810 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":142,"threads_started":1}
W20260926 05:47:28.960116 21441 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:28.954693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:51729 (local address 127.11.247.62:41141)
0926 05:47:28.954828 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:28.954830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:28.954840 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:28.954914 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:28.954914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:28.954926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:28.955004 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:28.955443 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.956016 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.956792 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:28.957009 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:28.957071 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:28.960036 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:51729: BlockingRecv error: recv got EOF from 127.11.247.1:51729 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":53,"threads_started":1}
Sep 26 05:47:29 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:29 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:29.092612 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:29.087465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:50457)
0926 05:47:29.087633 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:29.087647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:29.087724 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:29.087892 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:29.087893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:29.088159 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:29.088217 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:29.088219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.088912 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.088913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:29.089484 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:29.089486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.089856 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.089856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:29.089906 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:29.090384 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:29.090389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:29.092535 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
W20260926 05:47:29.092824 21441 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:29.087532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:50457 (local address 127.11.247.62:41141)
0926 05:47:29.087589 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:29.087592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:29.087732 (+ 140us) server_negotiation.cc:407] Connection header received
0926 05:47:29.087786 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:29.087787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:29.087809 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:29.087857 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:29.088300 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.088809 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.089572 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.089780 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.089837 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:29.092753 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:50457: BlockingRecv error: recv got EOF from 127.11.247.2:50457 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 05:47:29 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:29 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:29.238535 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:29.233429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:51697)
0926 05:47:29.233743 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:29.233753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:29.233846 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:29.233983 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:29.233983 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:29.234184 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:29.234232 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:29.234234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.235041 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.235042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:29.235591 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:29.235593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.236009 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.236010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:29.236079 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:29.236614 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:29.236619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:29.238435 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":173,"threads_started":1}
W20260926 05:47:29.238644 21441 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:29.233578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:51697 (local address 127.11.247.62:41141)
0926 05:47:29.233659 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:29.233661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:29.233821 (+ 160us) server_negotiation.cc:407] Connection header received
0926 05:47:29.233901 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:29.233902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:29.233917 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:29.233968 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:29.234328 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.234953 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.235700 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.235919 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.236005 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:29.238532 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:51697: BlockingRecv error: recv got EOF from 127.11.247.3:51697 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 05:47:29 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:29 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:29.966405 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:29.960736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:57847)
0926 05:47:29.960885 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:29.960902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:29.960958 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:29.961299 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:29.961300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:29.961499 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:29.961551 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:29.961553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.962430 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.962431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:29.963329 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:29.963333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.963801 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.963802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:29.963865 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:29.964379 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:29.964385 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:29.966301 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260926 05:47:29.966490 21445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:29.960861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:57847 (local address 127.11.247.62:41141)
0926 05:47:29.961000 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:29.961002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:29.961012 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:29.961123 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:29.961124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:29.961137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:29.961262 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:29.961674 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.962253 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.963451 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:29.963679 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:29.963799 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:29.966414 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:57847: BlockingRecv error: recv got EOF from 127.11.247.1:57847 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
Sep 26 05:47:30 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:30 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:30.098978 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:30.093196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:59451)
0926 05:47:30.093356 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:30.093370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:30.093438 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:30.093679 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:30.093681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:30.093850 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:30.093903 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:30.093905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.095111 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.095113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:30.095721 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:30.095723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.096205 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.096206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:30.096260 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:30.096818 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:30.096823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:30.098917 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
W20260926 05:47:30.099174 21445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:30.093406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:59451 (local address 127.11.247.62:41141)
0926 05:47:30.093484 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:30.093486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:30.093496 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:30.093531 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:30.093532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:30.093542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:30.093594 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:30.094025 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.094989 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.095805 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.096043 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.096107 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:30.099097 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:59451: BlockingRecv error: recv got EOF from 127.11.247.2:59451 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 05:47:30 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:30 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:30.244776 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:30.239250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:41591)
0926 05:47:30.239469 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:30.239482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:30.239595 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:30.239790 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:30.239790 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:30.239996 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:30.240046 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:30.240049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.240797 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.240799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:30.241479 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:30.241482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.241870 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.241871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:30.241931 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:30.242439 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:30.242444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:30.244715 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
W20260926 05:47:30.244925 21445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:30.239364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:41591 (local address 127.11.247.62:41141)
0926 05:47:30.239464 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:30.239466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:30.239571 (+ 105us) server_negotiation.cc:407] Connection header received
0926 05:47:30.239677 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:30.239678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:30.239694 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:30.239766 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:30.240142 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.240688 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.241576 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.241778 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.241868 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:30.244838 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:41591: BlockingRecv error: recv got EOF from 127.11.247.3:41591 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 05:47:30 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:30 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:30.979779 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:30.967178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:54371)
0926 05:47:30.967370 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:30.967382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:30.967459 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:30.967650 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:30.967651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:30.967821 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:30.967873 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:30.967874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.968707 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.968708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:30.969262 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:30.969264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.969742 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.969743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:30.969793 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:30.977378 (+ 7585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:30.977393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:30.979712 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":133,"threads_started":1}
W20260926 05:47:30.979976 21449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:30.967267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:54371 (local address 127.11.247.62:41141)
0926 05:47:30.967437 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:30.967439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:30.967451 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:47:30.967513 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:30.967514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:30.967525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:30.967630 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:30.967995 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.968597 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.969379 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:30.969651 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:30.969724 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:30.979889 (+ 10165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:54371: BlockingRecv error: recv got EOF from 127.11.247.1:54371 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
Sep 26 05:47:31 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:31 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:31.105701 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:31.099596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:59337)
0926 05:47:31.099936 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:31.099952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:31.100075 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:31.100288 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:31.100289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:31.100522 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:31.100577 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:31.100579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.101414 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.101416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:31.102081 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:31.102084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.102518 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.102519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:31.102662 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:31.103205 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:31.103210 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:31.105604 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":65,"threads_started":1}
W20260926 05:47:31.105868 21449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:31.099862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:59337 (local address 127.11.247.62:41141)
0926 05:47:31.099955 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:31.099958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:31.100051 (+ 93us) server_negotiation.cc:407] Connection header received
0926 05:47:31.100168 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:31.100169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:31.100184 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:31.100264 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:31.100683 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.101312 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.102189 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.102412 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.102499 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:31.105766 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:59337: BlockingRecv error: recv got EOF from 127.11.247.2:59337 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 05:47:31 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:31 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.3@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:31.251052 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:31.245378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.3:49947)
0926 05:47:31.245531 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:31.245542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:31.245648 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:31.245824 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:31.245825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:31.246016 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:31.246066 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:31.246068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.246879 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.246880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:31.247578 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:31.247581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.248018 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.248019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:31.248072 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:31.248679 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:31.248684 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:31.250986 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
W20260926 05:47:31.251276 21449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:31.245561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:49947 (local address 127.11.247.62:41141)
0926 05:47:31.245669 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:31.245671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:31.245682 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:31.245731 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:31.245733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:31.245747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:31.245814 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:31.246163 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.246791 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.247697 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.247931 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.247997 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:31.251192 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.3:49947: BlockingRecv error: recv got EOF from 127.11.247.3:49947 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 05:47:31 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:31 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.1@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:31.985868 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:31.980511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.1:40879)
0926 05:47:31.980676 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:31.980687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:31.980747 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:31.980981 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:31.980982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:31.981153 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:31.981211 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:31.981212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.981917 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.981918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:31.982494 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:31.982497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.982915 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.982916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:31.982967 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:31.983563 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:31.983574 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:31.985803 (+ 2229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260926 05:47:31.986106 21453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:31.980627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:40879 (local address 127.11.247.62:41141)
0926 05:47:31.980804 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:31.980806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:31.980816 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:47:31.980844 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:31.980845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:31.980856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:31.980933 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:31.981291 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.981813 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.982628 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:31.982836 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:31.982895 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:31.986032 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.1:40879: BlockingRecv error: recv got EOF from 127.11.247.1:40879 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
Sep 26 05:47:32 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:47:32 dist-test-slave-21p4 krb5kdc[20394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.247.2@KRBTEST.COM for oryx/127.11.247.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:47:32.111827 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.106389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:41141 (local address 127.11.247.2:60457)
0926 05:47:32.106601 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:32.106614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:32.106682 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:32.106917 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:32.106918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:32.107183 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:32.107253 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.107256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.108045 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.108046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.108604 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.108606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.109023 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.109024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.109067 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.109574 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:32.109579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:32.111737 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.247.62:41141: Server oryx/127.11.247.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":117,"threads_started":1}
W20260926 05:47:32.112069 21453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:32.106447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:60457 (local address 127.11.247.62:41141)
0926 05:47:32.106526 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.106528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.106669 (+ 141us) server_negotiation.cc:407] Connection header received
0926 05:47:32.106777 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.106793 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.106809 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.106885 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.107350 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.107921 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.108713 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.108940 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.109012 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.111965 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.247.2:60457: BlockingRecv error: recv got EOF from 127.11.247.2:60457 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:47:32.162241 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20977
I20260926 05:47:32.167551 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21129
I20260926 05:47:32.172256 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21263
I20260926 05:47:32.177079 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 20907
2026-09-26T05:47:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17565 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T05:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:32Z Disabled control of system clock
I20260926 05:47:32.208353 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:36825
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:37599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 05:47:32.296156 21458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:32.296408 21458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:32.296430 21458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:32.297663 21458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:32.297709 21458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:32.297722 21458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:32.297736 21458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:32.297748 21458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:32.299160 21458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:36825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:32.299448 21458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:32.299729 21458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:32.305178 21464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.305137 21466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.305195 21463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.305372 21458 server_base.cc:1034] running on GCE node
I20260926 05:47:32.305660 21458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:32.306015 21458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:32.307267 21458 hybrid_clock.cc:648] HybridClock initialized: now 1790401652307211 us; error 76 us; skew 500 ppm
I20260926 05:47:32.309386 21458 webserver.cc:460] Webserver started at http://127.11.247.62:33321/ using document root <none> and password file <none>
I20260926 05:47:32.309736 21458 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:32.309824 21458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:32.309988 21458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:32.311363 21458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "fb98bfbe4c7944c69f0967752f4a00ce"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "bd3b13e0d0b4c1988eb80c08e2064c11"
server_key_iv: "9824ad824b79c4f4164c3ca8b96edafd"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.311784 21458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "fb98bfbe4c7944c69f0967752f4a00ce"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "bd3b13e0d0b4c1988eb80c08e2064c11"
server_key_iv: "9824ad824b79c4f4164c3ca8b96edafd"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.313331 21458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:32.314229 21472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.314539 21458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:32.314682 21458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "fb98bfbe4c7944c69f0967752f4a00ce"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "bd3b13e0d0b4c1988eb80c08e2064c11"
server_key_iv: "9824ad824b79c4f4164c3ca8b96edafd"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.314769 21458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:32.323093 21458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:32.323458 21458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:32.323622 21458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:32.327936 21458 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:36825
I20260926 05:47:32.327970 21524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:36825 every 8 connection(s)
I20260926 05:47:32.328363 21458 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:32.329043 21525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:32.331846 21525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce: Bootstrap starting.
I20260926 05:47:32.332477 21525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:32.332792 21525 log.cc:824] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:32.333496 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21458
I20260926 05:47:32.333642 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:32.333665 21525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce: No bootstrap required, opened a new log
I20260926 05:47:32.333858 12252 external_mini_cluster.cc:1428] Setting key 971139cafa9eebb2a4922622c82c663b
I20260926 05:47:32.335718 21525 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36825 } }
I20260926 05:47:32.335902 21525 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:32.335954 21525 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb98bfbe4c7944c69f0967752f4a00ce, State: Initialized, Role: FOLLOWER
I20260926 05:47:32.336160 21525 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36825 } }
I20260926 05:47:32.336267 21525 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:32.336311 21525 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:32.336390 21525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:32.337002 21525 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36825 } }
I20260926 05:47:32.337095 21525 leader_election.cc:302] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb98bfbe4c7944c69f0967752f4a00ce; no voters:
I20260926 05:47:32.337287 21525 leader_election.cc:288] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:32.337347 21530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:32.337563 21530 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [term 1 LEADER]: Becoming Leader. State: Replica: fb98bfbe4c7944c69f0967752f4a00ce, State: Running, Role: LEADER
I20260926 05:47:32.337591 21525 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:32.337664 21530 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36825 } }
I20260926 05:47:32.337971 21530 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb98bfbe4c7944c69f0967752f4a00ce. Latest consensus state: current_term: 1 leader_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36825 } } }
I20260926 05:47:32.338052 21530 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:32.337983 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb98bfbe4c7944c69f0967752f4a00ce" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 36825 } } }
I20260926 05:47:32.338245 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:32.338284 21536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:32.338855 21536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:32.340425 21547 mvcc.cc:205] Tried to move back new op lower bound from 7333485167984132096 to 7333485167975374848. Current Snapshot: MvccSnapshot[applied={T|T < 7333485167984132096}]
I20260926 05:47:32.340946 21536 catalog_manager.cc:1347] Generated new cluster ID: 235a3600cce5411193c439c0d1274bbf
I20260926 05:47:32.341009 21536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:32.341984 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.334502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.11.247.62:36825)
0926 05:47:32.334728 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.334736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.334766 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:47:32.335703 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.335712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.335796 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.336037 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.336345 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.338061 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.339335 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.340084 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.340592 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.341091 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:32.341095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:32.341095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:32.341314 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:32.341354 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:32.341369 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:32.341658 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:32.341690 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 05:47:32.341771 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260926 05:47:32.363384 21536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:32.364184 21536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:32.371443 21536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb98bfbe4c7944c69f0967752f4a00ce: Generated new TSK 0
I20260926 05:47:32.371697 21536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:47:32.376271 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36825
--builtin_ntp_servers=127.11.247.20:37599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 05:47:32.465955 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:32.466243 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:32.466282 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:32.467674 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:32.467744 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:32.467789 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:32.469481 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:32.469870 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:32.470534 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:32.477437 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.477448 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.477437 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.477659 21549 server_base.cc:1034] running on GCE node
I20260926 05:47:32.478025 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:32.478305 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:32.479499 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790401652479472 us; error 44 us; skew 500 ppm
I20260926 05:47:32.480918 21549 webserver.cc:460] Webserver started at http://127.11.247.1:40223/ using document root <none> and password file <none>
I20260926 05:47:32.481235 21549 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:32.481314 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:32.481472 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:32.482715 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "b405c0691438477087dacf70df9f9fa5"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "fa550ddd4e46c73324ced8b1a4906dc4"
server_key_iv: "40154ad9fd763c2bc9c278178884248d"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.483124 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "b405c0691438477087dacf70df9f9fa5"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "fa550ddd4e46c73324ced8b1a4906dc4"
server_key_iv: "40154ad9fd763c2bc9c278178884248d"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.484704 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:32.485726 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.486106 21549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:32.486248 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "b405c0691438477087dacf70df9f9fa5"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "fa550ddd4e46c73324ced8b1a4906dc4"
server_key_iv: "40154ad9fd763c2bc9c278178884248d"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.486346 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:32.506398 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:32.506884 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:32.507064 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:32.507320 21549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:32.507704 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:32.507763 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.507809 21549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:32.507841 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.514765 21549 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:41685
I20260926 05:47:32.514806 21676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:41685 every 8 connection(s)
I20260926 05:47:32.515197 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:32.521137 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.516028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:40159 (local address 127.11.247.62:36825)
0926 05:47:32.516108 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.516110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.516361 (+ 251us) server_negotiation.cc:407] Connection header received
0926 05:47:32.516899 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.516900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.516913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.517018 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.518063 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.518709 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.519764 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.519970 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.520027 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.520868 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:32.520869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:32.520870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:32.520875 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:32.520890 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:32.520895 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:32.520938 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:32.521037 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 05:47:32.521068 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:47:32.521409 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.516046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:36825 (local address 127.11.247.1:40159)
0926 05:47:32.516232 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:32.516271 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:32.516797 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:32.517206 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:32.517211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:32.517222 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:32.517923 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.517930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.518849 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.518850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.519678 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.519680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.520066 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.520067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.520194 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.520726 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:32.520726 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:32.520756 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:32.520780 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:32.520952 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:32.520957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:32.520959 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:47:32.521013 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 05:47:32.521133 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260926 05:47:32.522080 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21549
I20260926 05:47:32.522043 21677 heartbeater.cc:347] Connected to a master server at 127.11.247.62:36825
I20260926 05:47:32.522186 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:32.522202 21677 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:32.522389 12252 external_mini_cluster.cc:1428] Setting key d07f27f7646ced190ee4f29b8eba47ee
I20260926 05:47:32.522406 21677 heartbeater.cc:511] Master 127.11.247.62:36825 requested a full tablet report, sending...
I20260926 05:47:32.522961 21489 ts_manager.cc:194] Registered new tserver with Master: b405c0691438477087dacf70df9f9fa5 (127.11.247.1:41685)
I20260926 05:47:32.523658 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36825
--builtin_ntp_servers=127.11.247.20:37599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 05:47:32.523949 21489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.1:40159
W20260926 05:47:32.609846 21680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:32.610117 21680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:32.610143 21680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:32.611711 21680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:32.611778 21680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:32.611809 21680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:32.613397 21680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:32.613759 21680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:32.614149 21680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:32.620296 21685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.620304 21686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.620443 21680 server_base.cc:1034] running on GCE node
W20260926 05:47:32.620582 21688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.620896 21680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:32.621217 21680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:32.622429 21680 hybrid_clock.cc:648] HybridClock initialized: now 1790401652622421 us; error 64 us; skew 500 ppm
I20260926 05:47:32.623976 21680 webserver.cc:460] Webserver started at http://127.11.247.2:46483/ using document root <none> and password file <none>
I20260926 05:47:32.624305 21680 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:32.624382 21680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:32.624539 21680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:32.625712 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "d3746e093429458380b8f435fdad96e9"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "07e255c440d8a489eaa457c1b587e5d2"
server_key_iv: "900265302ebc20868a3e26a0811abade"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.626111 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "d3746e093429458380b8f435fdad96e9"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "07e255c440d8a489eaa457c1b587e5d2"
server_key_iv: "900265302ebc20868a3e26a0811abade"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.627637 21680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:47:32.628558 21694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.628753 21680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:32.628861 21680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "d3746e093429458380b8f435fdad96e9"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "07e255c440d8a489eaa457c1b587e5d2"
server_key_iv: "900265302ebc20868a3e26a0811abade"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.628947 21680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:32.649276 21680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:32.649704 21680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:32.649901 21680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:32.650203 21680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:32.650683 21680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:32.650745 21680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.650792 21680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:32.650830 21680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.657796 21680 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35587
I20260926 05:47:32.657845 21807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35587 every 8 connection(s)
I20260926 05:47:32.658288 21680 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:47:32.659417 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21680
I20260926 05:47:32.659536 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:32.659720 12252 external_mini_cluster.cc:1428] Setting key 2dc87fee6af28ea3c08e7deb9fadcff8
I20260926 05:47:32.660935 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36825
--builtin_ntp_servers=127.11.247.20:37599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 05:47:32.666678 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.659262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:41913 (local address 127.11.247.62:36825)
0926 05:47:32.659352 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.659355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.660201 (+ 846us) server_negotiation.cc:407] Connection header received
0926 05:47:32.660950 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.660951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.660965 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.661055 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.661961 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.662946 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.664787 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.665036 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.665100 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.666299 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:32.666301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:32.666301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:32.666307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:32.666323 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:32.666327 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:32.666376 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:32.666518 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 05:47:32.666549 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 05:47:32.667068 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.659221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:36825 (local address 127.11.247.2:41913)
0926 05:47:32.660025 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:32.660081 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:32.660800 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:32.661187 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:32.661193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:32.661205 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:32.661848 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.661853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.663068 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.663069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.664645 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.664649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.665148 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.665149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.665372 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.666085 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:32.666086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:32.666136 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:32.666177 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:32.666406 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:32.666415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:32.666419 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:32.666516 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:47:32.666717 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":99,"threads_started":1}
I20260926 05:47:32.667796 21808 heartbeater.cc:347] Connected to a master server at 127.11.247.62:36825
I20260926 05:47:32.667912 21808 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:32.668141 21808 heartbeater.cc:511] Master 127.11.247.62:36825 requested a full tablet report, sending...
I20260926 05:47:32.668592 21489 ts_manager.cc:194] Registered new tserver with Master: d3746e093429458380b8f435fdad96e9 (127.11.247.2:35587)
I20260926 05:47:32.669248 21489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.2:41913
W20260926 05:47:32.757874 21811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:32.758126 21811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:32.758150 21811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:32.759523 21811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:32.759574 21811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:32.759603 21811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:32.761163 21811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:37599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:32.761389 21811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:32.761680 21811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:32.767516 21817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.767519 21816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.767549 21811 server_base.cc:1034] running on GCE node
W20260926 05:47:32.767516 21819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.768060 21811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:32.768373 21811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:32.769544 21811 hybrid_clock.cc:648] HybridClock initialized: now 1790401652769529 us; error 28 us; skew 500 ppm
I20260926 05:47:32.771020 21811 webserver.cc:460] Webserver started at http://127.11.247.3:37407/ using document root <none> and password file <none>
I20260926 05:47:32.771296 21811 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:32.771346 21811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:32.771524 21811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:32.772668 21811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "a98d0941309841459b3b355d6f001be5"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "5e0f3d2ec52e539097db36c76b8898ad"
server_key_iv: "101f6d5bad5d88293c068b5925acc3c2"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.773057 21811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "a98d0941309841459b3b355d6f001be5"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "5e0f3d2ec52e539097db36c76b8898ad"
server_key_iv: "101f6d5bad5d88293c068b5925acc3c2"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.774495 21811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:47:32.775544 21825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.775748 21811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:32.775856 21811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "a98d0941309841459b3b355d6f001be5"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "5e0f3d2ec52e539097db36c76b8898ad"
server_key_iv: "101f6d5bad5d88293c068b5925acc3c2"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.775929 21811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:32.789319 21811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:32.789681 21811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:32.789806 21811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:32.790069 21811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:32.790510 21811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:32.790551 21811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.790655 21811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:32.790676 21811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.797286 21811 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:45313
I20260926 05:47:32.797392 21938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:45313 every 8 connection(s)
I20260926 05:47:32.797742 21811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:32.803351 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.798560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:58937 (local address 127.11.247.62:36825)
0926 05:47:32.798656 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.798659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.798865 (+ 206us) server_negotiation.cc:407] Connection header received
0926 05:47:32.799471 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.799473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.799485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.799543 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.800250 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.800772 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.801793 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.802001 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.802090 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.803035 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:32.803037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:32.803037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:32.803042 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:32.803057 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:32.803062 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:32.803134 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:32.803231 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 05:47:32.803264 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 05:47:32.803574 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.798501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:36825 (local address 127.11.247.3:58937)
0926 05:47:32.798739 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:32.798774 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:32.799349 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:32.799634 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:32.799645 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:32.799655 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:32.800148 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.800152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.800876 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.800877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.801692 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:32.801694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:32.802098 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:32.802099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:32.802242 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:32.802884 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:32.802884 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:32.802913 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:32.802937 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:32.803133 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:32.803140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:32.803143 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:32.803233 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:47:32.803338 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":131,"threads_started":1}
I20260926 05:47:32.804142 21939 heartbeater.cc:347] Connected to a master server at 127.11.247.62:36825
I20260926 05:47:32.804236 21939 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:32.804384 21939 heartbeater.cc:511] Master 127.11.247.62:36825 requested a full tablet report, sending...
I20260926 05:47:32.804756 21489 ts_manager.cc:194] Registered new tserver with Master: a98d0941309841459b3b355d6f001be5 (127.11.247.3:45313)
I20260926 05:47:32.805104 21489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.3:58937
I20260926 05:47:32.807147 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21811
I20260926 05:47:32.807252 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:32.807531 12252 external_mini_cluster.cc:1428] Setting key 74251704ef0479babdf11ced41a2b287
I20260926 05:47:32.808498 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:47:32.810637 21529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:32.810013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.11.247.62:36825)
0926 05:47:32.810118 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.810121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.810221 (+ 100us) server_negotiation.cc:407] Connection header received
0926 05:47:32.810305 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.810306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.810320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.810390 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.810517 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41172: BlockingRecv error: recv got EOF from 127.0.0.1:41172 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 05:47:32.810734 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21549
I20260926 05:47:32.815634 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21680
I20260926 05:47:32.820716 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21811
I20260926 05:47:32.825723 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21458
2026-09-26T05:47:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (645 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T05:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:32Z Disabled control of system clock
I20260926 05:47:32.854559 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:45203
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:47:32.940577 21950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:32.940810 21950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:32.940834 21950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:32.942135 21950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:32.942198 21950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:32.942210 21950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:32.942224 21950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:32.942236 21950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:32.943830 21950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:45203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:32.944152 21950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:32.944469 21950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:32.949913 21958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.950006 21956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:32.949975 21955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:32.950145 21950 server_base.cc:1034] running on GCE node
I20260926 05:47:32.950436 21950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:32.950781 21950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:32.951992 21950 hybrid_clock.cc:648] HybridClock initialized: now 1790401652951966 us; error 33 us; skew 500 ppm
I20260926 05:47:32.953521 21950 webserver.cc:460] Webserver started at http://127.11.247.62:44111/ using document root <none> and password file <none>
I20260926 05:47:32.953826 21950 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:32.953899 21950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:32.954105 21950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:32.955443 21950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "478bf74c618d4f24a5eec460906493e6"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "d49d1c29ddc944179829071907b75546"
server_key_iv: "c567445620e8a523dcf8bd423975b002"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.955852 21950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "478bf74c618d4f24a5eec460906493e6"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "d49d1c29ddc944179829071907b75546"
server_key_iv: "c567445620e8a523dcf8bd423975b002"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.957540 21950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:47:32.958528 21964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:32.958755 21950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:32.958871 21950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "478bf74c618d4f24a5eec460906493e6"
format_stamp: "Formatted at 2026-09-26 05:47:32 on dist-test-slave-21p4"
server_key: "d49d1c29ddc944179829071907b75546"
server_key_iv: "c567445620e8a523dcf8bd423975b002"
server_key_version: "encryptionkey@0"
I20260926 05:47:32.958945 21950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:32.962445 21950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:32.962750 21950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:32.962931 21950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:32.966965 21950 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:45203
I20260926 05:47:32.966998 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:45203 every 8 connection(s)
I20260926 05:47:32.967374 21950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:32.968113 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:32.969664 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 21950
I20260926 05:47:32.969780 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:32.970050 12252 external_mini_cluster.cc:1428] Setting key feb73603f7e36e3db2032d332d9d7f6c
I20260926 05:47:32.972749 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6: Bootstrap starting.
I20260926 05:47:32.973567 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:32.970799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.11.247.62:45203)
0926 05:47:32.971036 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:32.971045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:32.971073 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:47:32.971557 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:32.971564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:32.971610 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:32.971745 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:32.973058 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:32.973063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:32.973063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:32.973067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:32.973111 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:32.973122 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:32.973197 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:32.973280 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:47:32.973368 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260926 05:47:32.973578 22017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:32.973968 22017 log.cc:824] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:32.974836 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6: No bootstrap required, opened a new log
I20260926 05:47:32.976282 22017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478bf74c618d4f24a5eec460906493e6" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45203 } }
I20260926 05:47:32.976473 22017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:32.976524 22017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478bf74c618d4f24a5eec460906493e6, State: Initialized, Role: FOLLOWER
I20260926 05:47:32.976683 22017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478bf74c618d4f24a5eec460906493e6" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45203 } }
I20260926 05:47:32.976783 22017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:32.976830 22017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:32.976890 22017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:32.977593 22017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478bf74c618d4f24a5eec460906493e6" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45203 } }
I20260926 05:47:32.977722 22017 leader_election.cc:302] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 478bf74c618d4f24a5eec460906493e6; no voters:
I20260926 05:47:32.977922 22017 leader_election.cc:288] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:32.978230 22017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:32.978317 22022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:32.978492 22022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [term 1 LEADER]: Becoming Leader. State: Replica: 478bf74c618d4f24a5eec460906493e6, State: Running, Role: LEADER
I20260926 05:47:32.978652 22022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478bf74c618d4f24a5eec460906493e6" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45203 } }
I20260926 05:47:32.979130 22024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478bf74c618d4f24a5eec460906493e6. Latest consensus state: current_term: 1 leader_uuid: "478bf74c618d4f24a5eec460906493e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478bf74c618d4f24a5eec460906493e6" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45203 } } }
I20260926 05:47:32.979254 22024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:32.979312 22023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "478bf74c618d4f24a5eec460906493e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478bf74c618d4f24a5eec460906493e6" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 45203 } } }
I20260926 05:47:32.979401 22023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:32.979491 22032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:32.980014 22032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:32.981693 22032 catalog_manager.cc:1347] Generated new cluster ID: c1855aff5c514602ba4dcd13a383ed9b
I20260926 05:47:32.981750 22032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:32.994939 22032 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:32.995208 22032 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:33.007476 22032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 478bf74c618d4f24a5eec460906493e6: Generated new TSK 0
I20260926 05:47:33.007737 22032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:47:33.011010 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45203
--builtin_ntp_servers=127.11.247.20:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:47:33.103408 22041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:33.103659 22041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:33.103685 22041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:33.105118 22041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:33.105180 22041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:33.105211 22041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:33.106935 22041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:33.107234 22041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:33.107590 22041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:33.114051 22047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:33.114180 22046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:33.114331 22041 server_base.cc:1034] running on GCE node
W20260926 05:47:33.114053 22049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:33.114697 22041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:33.114997 22041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:33.116194 22041 hybrid_clock.cc:648] HybridClock initialized: now 1790401653116178 us; error 35 us; skew 500 ppm
I20260926 05:47:33.117733 22041 webserver.cc:460] Webserver started at http://127.11.247.1:35071/ using document root <none> and password file <none>
I20260926 05:47:33.118026 22041 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:33.118149 22041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:33.118319 22041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:33.119575 22041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "8c89c895af0d4bde940ce605476d952c"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "b30dfb6b180ce3404ec5b20653715caf"
server_key_iv: "7244e4db9047fb565dc0104c01dc148c"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.119968 22041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "8c89c895af0d4bde940ce605476d952c"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "b30dfb6b180ce3404ec5b20653715caf"
server_key_iv: "7244e4db9047fb565dc0104c01dc148c"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.121480 22041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:47:33.122341 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.122730 22041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:33.122836 22041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "8c89c895af0d4bde940ce605476d952c"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "b30dfb6b180ce3404ec5b20653715caf"
server_key_iv: "7244e4db9047fb565dc0104c01dc148c"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.122920 22041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:33.127671 22041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:33.127940 22041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:33.128046 22041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:33.128217 22041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:33.128611 22041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:33.128651 22041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.128676 22041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:33.128690 22041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.135085 22041 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:46625
I20260926 05:47:33.135191 22168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:46625 every 8 connection(s)
I20260926 05:47:33.135543 22041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:33.137243 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22041
I20260926 05:47:33.137326 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:33.137568 12252 external_mini_cluster.cc:1428] Setting key 9927d1413226c96a64ef982c795b7685
I20260926 05:47:33.138820 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45203
--builtin_ntp_servers=127.11.247.20:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 05:47:33.139767 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:33.136523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:47835 (local address 127.11.247.62:45203)
0926 05:47:33.136601 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:33.136603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:33.136894 (+ 291us) server_negotiation.cc:407] Connection header received
0926 05:47:33.137561 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:33.137562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:33.137569 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:33.137656 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:33.139372 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:33.139374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:33.139374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:33.139380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:33.139409 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:33.139415 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:33.139485 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:33.139645 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 05:47:33.139663 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:47:33.140033 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:33.136506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45203 (local address 127.11.247.1:47835)
0926 05:47:33.136766 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:33.136802 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:33.137410 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:33.137800 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:33.137805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:33.137815 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:33.139195 (+ 1380us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:33.139195 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:33.139231 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:33.139265 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:33.139537 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:33.139545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:33.139548 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:33.139635 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:47:33.139701 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":129,"threads_started":1}
I20260926 05:47:33.140606 22169 heartbeater.cc:347] Connected to a master server at 127.11.247.62:45203
I20260926 05:47:33.140720 22169 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:33.140908 22169 heartbeater.cc:511] Master 127.11.247.62:45203 requested a full tablet report, sending...
I20260926 05:47:33.141544 21981 ts_manager.cc:194] Registered new tserver with Master: 8c89c895af0d4bde940ce605476d952c (127.11.247.1:46625)
W20260926 05:47:33.225114 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:33.225346 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:33.225373 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:33.226883 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:33.226951 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:33.227000 22172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:33.228624 22172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:33.228924 22172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:33.229283 22172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:33.234766 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:33.234766 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:33.234802 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:33.235255 22172 server_base.cc:1034] running on GCE node
I20260926 05:47:33.235544 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:33.235828 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:33.237026 22172 hybrid_clock.cc:648] HybridClock initialized: now 1790401653237009 us; error 27 us; skew 500 ppm
I20260926 05:47:33.238673 22172 webserver.cc:460] Webserver started at http://127.11.247.2:45077/ using document root <none> and password file <none>
I20260926 05:47:33.238981 22172 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:33.239066 22172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:33.239257 22172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:33.240479 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "523531c7da704f6097c97941941a45e5"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "e93a3e09aa22315e9819c60fa8a37efa"
server_key_iv: "9fb28c8072d05b9fd1397dcf4e47e5dd"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.240908 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "523531c7da704f6097c97941941a45e5"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "e93a3e09aa22315e9819c60fa8a37efa"
server_key_iv: "9fb28c8072d05b9fd1397dcf4e47e5dd"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.242762 22172 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 05:47:33.243880 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.244263 22172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:47:33.244376 22172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "523531c7da704f6097c97941941a45e5"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "e93a3e09aa22315e9819c60fa8a37efa"
server_key_iv: "9fb28c8072d05b9fd1397dcf4e47e5dd"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.244448 22172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:33.248092 22172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:33.248344 22172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:33.248445 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:33.248610 22172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:33.248991 22172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:33.249037 22172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.249058 22172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:33.249094 22172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.255682 22172 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:43223
I20260926 05:47:33.255767 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:43223 every 8 connection(s)
I20260926 05:47:33.256125 22172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:47:33.259691 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:33.256970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:48943 (local address 127.11.247.62:45203)
0926 05:47:33.257027 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:33.257030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:33.257253 (+ 223us) server_negotiation.cc:407] Connection header received
0926 05:47:33.257845 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:33.257847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:33.257855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:33.257920 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:33.259367 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:33.259369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:33.259369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:33.259375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:33.259394 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:33.259399 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:33.259463 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:33.259586 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 05:47:33.259605 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:47:33.259930 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:33.256929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45203 (local address 127.11.247.2:48943)
0926 05:47:33.257120 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:33.257158 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:33.257718 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:33.258009 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:33.258015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:33.258027 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:33.259181 (+ 1154us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:33.259181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:33.259215 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:33.259251 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:33.259472 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:33.259478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:33.259481 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:47:33.259599 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 05:47:33.259669 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260926 05:47:33.260596 22300 heartbeater.cc:347] Connected to a master server at 127.11.247.62:45203
I20260926 05:47:33.260699 22300 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:33.260887 22300 heartbeater.cc:511] Master 127.11.247.62:45203 requested a full tablet report, sending...
I20260926 05:47:33.261365 21981 ts_manager.cc:194] Registered new tserver with Master: 523531c7da704f6097c97941941a45e5 (127.11.247.2:43223)
I20260926 05:47:33.265210 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22172
I20260926 05:47:33.265304 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:33.265534 12252 external_mini_cluster.cc:1428] Setting key c310142380081b74b233ec25828954d0
I20260926 05:47:33.266556 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45203
--builtin_ntp_servers=127.11.247.20:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:47:33.352941 22303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:33.353165 22303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:33.353190 22303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:33.354508 22303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:33.354554 22303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:33.354640 22303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:33.356086 22303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:45203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:33.356369 22303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:33.356694 22303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:33.362015 22309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:33.362000 22308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:33.362054 22311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:33.362275 22303 server_base.cc:1034] running on GCE node
I20260926 05:47:33.362565 22303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:33.362890 22303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:33.364087 22303 hybrid_clock.cc:648] HybridClock initialized: now 1790401653364068 us; error 48 us; skew 500 ppm
I20260926 05:47:33.365372 22303 webserver.cc:460] Webserver started at http://127.11.247.3:34741/ using document root <none> and password file <none>
I20260926 05:47:33.365669 22303 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:33.365749 22303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:33.365916 22303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:33.367130 22303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "b88ac2e694f949a68c8b85d457d522ac"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "cbb06aa8ceba967c12d4d88089d7d46e"
server_key_iv: "be3c129d4dff42fa49ab69f6387d2d0e"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.367501 22303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "b88ac2e694f949a68c8b85d457d522ac"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "cbb06aa8ceba967c12d4d88089d7d46e"
server_key_iv: "be3c129d4dff42fa49ab69f6387d2d0e"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.368848 22303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:33.369668 22317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.369853 22303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:33.369952 22303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "b88ac2e694f949a68c8b85d457d522ac"
format_stamp: "Formatted at 2026-09-26 05:47:33 on dist-test-slave-21p4"
server_key: "cbb06aa8ceba967c12d4d88089d7d46e"
server_key_iv: "be3c129d4dff42fa49ab69f6387d2d0e"
server_key_version: "encryptionkey@0"
I20260926 05:47:33.370077 22303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:33.373570 22303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:33.373867 22303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:33.374014 22303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:33.374231 22303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:33.374557 22303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:33.374635 22303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.374675 22303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:33.374701 22303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:33.380960 22303 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:39121
I20260926 05:47:33.381047 22430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:39121 every 8 connection(s)
I20260926 05:47:33.381397 22303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:33.381805 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22303
I20260926 05:47:33.381901 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:33.382121 12252 external_mini_cluster.cc:1428] Setting key e19a4082e490bc5638fef2aaa3fdfe44
I20260926 05:47:33.385244 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:33.382317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:52829 (local address 127.11.247.62:45203)
0926 05:47:33.382400 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:33.382403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:33.382770 (+ 367us) server_negotiation.cc:407] Connection header received
0926 05:47:33.383456 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:33.383457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:33.383464 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:33.383516 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:33.384912 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:33.384913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:33.384914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:47:33.384919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:33.384935 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:47:33.384940 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:33.385035 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:33.385148 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:47:33.385166 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:47:33.385478 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:33.382261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:45203 (local address 127.11.247.3:52829)
0926 05:47:33.382638 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:33.382687 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:33.383337 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:33.383600 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:33.383604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:33.383615 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:33.384722 (+ 1107us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:47:33.384723 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:33.384755 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:47:33.384787 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:33.385038 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:33.385045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:33.385049 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:33.385160 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:47:33.385230 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260926 05:47:33.386022 22431 heartbeater.cc:347] Connected to a master server at 127.11.247.62:45203
I20260926 05:47:33.386133 22431 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:33.386304 22431 heartbeater.cc:511] Master 127.11.247.62:45203 requested a full tablet report, sending...
I20260926 05:47:33.386744 21981 ts_manager.cc:194] Registered new tserver with Master: b88ac2e694f949a68c8b85d457d522ac (127.11.247.3:39121)
I20260926 05:47:33.394435 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:47:33.396636 22020 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:33.395996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.11.247.62:45203)
0926 05:47:33.396077 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:33.396080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:33.396189 (+ 109us) server_negotiation.cc:407] Connection header received
0926 05:47:33.396288 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:33.396289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:33.396302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:33.396374 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:33.396544 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57396: BlockingRecv error: recv got EOF from 127.0.0.1:57396 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:47:33.396761 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22041
I20260926 05:47:33.402230 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22172
I20260926 05:47:33.407223 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22303
I20260926 05:47:33.412015 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 21950
2026-09-26T05:47:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (587 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:33 dist-test-slave-21p4 krb5kdc[22441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:33 dist-test-slave-21p4 krb5kdc[22441](info): set up 2 sockets
Sep 26 05:47:33 dist-test-slave-21p4 krb5kdc[22441](info): commencing operation
krb5kdc: starting...
W20260926 05:47:34.483724 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:34 dist-test-slave-21p4 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:47:34.631808 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37643
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:34.733868 22454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:34.734350 22454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:34.734395 22454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:34.736579 22454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:47:34.736666 22454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:34.736688 22454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:34.736707 22454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:47:34.736725 22454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:47:34.739205 22454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:34.739553 22454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:34.739820 22454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:34.747273 22459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:34.747323 22462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:34.750713 22454 server_base.cc:1034] running on GCE node
W20260926 05:47:34.750798 22460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:34.751187 22454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:34.751566 22454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:34.752768 22454 hybrid_clock.cc:648] HybridClock initialized: now 1790401654752752 us; error 58 us; skew 500 ppm
Sep 26 05:47:34 dist-test-slave-21p4 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401654, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:34.755972 22454 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:47:34.756575 22454 webserver.cc:460] Webserver started at http://127.11.247.62:43529/ using document root <none> and password file <none>
I20260926 05:47:34.756863 22454 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:34.756945 22454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:34.757118 22454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:34.758460 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "f8e57311c3c4450eb776afcab2159647"
format_stamp: "Formatted at 2026-09-26 05:47:34 on dist-test-slave-21p4"
server_key: "0ce871ef799ea90cfac534e69df0c493"
server_key_iv: "19125d7b6068abcab3ff337811d78447"
server_key_version: "encryptionkey@0"
I20260926 05:47:34.758936 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "f8e57311c3c4450eb776afcab2159647"
format_stamp: "Formatted at 2026-09-26 05:47:34 on dist-test-slave-21p4"
server_key: "0ce871ef799ea90cfac534e69df0c493"
server_key_iv: "19125d7b6068abcab3ff337811d78447"
server_key_version: "encryptionkey@0"
I20260926 05:47:34.760501 22454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:34.761385 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:34.761734 22454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:34.761857 22454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "f8e57311c3c4450eb776afcab2159647"
format_stamp: "Formatted at 2026-09-26 05:47:34 on dist-test-slave-21p4"
server_key: "0ce871ef799ea90cfac534e69df0c493"
server_key_iv: "19125d7b6068abcab3ff337811d78447"
server_key_version: "encryptionkey@0"
I20260926 05:47:34.761981 22454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:34.785318 22454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:34.788066 22454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:34.788532 22454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:34.793598 22454 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:37643
I20260926 05:47:34.793725 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:37643 every 8 connection(s)
I20260926 05:47:34.794178 22454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:47:34.796087 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:34.798422 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22454
I20260926 05:47:34.798553 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:47:34.798784 12252 external_mini_cluster.cc:1428] Setting key 26c25bc553b48326d0ef1eccb7daeeb9
I20260926 05:47:34.799299 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647: Bootstrap starting.
I20260926 05:47:34.800065 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:34.800387 22522 log.cc:824] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:34.801162 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647: No bootstrap required, opened a new log
I20260926 05:47:34.803716 22522 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e57311c3c4450eb776afcab2159647" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37643 } }
I20260926 05:47:34.803939 22522 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:34.803974 22522 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e57311c3c4450eb776afcab2159647, State: Initialized, Role: FOLLOWER
I20260926 05:47:34.804147 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e57311c3c4450eb776afcab2159647" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37643 } }
I20260926 05:47:34.804245 22522 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:47:34.804279 22522 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:47:34.804330 22522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:34.805220 22522 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e57311c3c4450eb776afcab2159647" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37643 } }
I20260926 05:47:34.805334 22522 leader_election.cc:302] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8e57311c3c4450eb776afcab2159647; no voters:
I20260926 05:47:34.805567 22522 leader_election.cc:288] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:47:34.805634 22527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:34.805830 22527 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [term 1 LEADER]: Becoming Leader. State: Replica: f8e57311c3c4450eb776afcab2159647, State: Running, Role: LEADER
I20260926 05:47:34.805912 22522 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:47:34.805961 22527 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e57311c3c4450eb776afcab2159647" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37643 } }
I20260926 05:47:34.806344 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8e57311c3c4450eb776afcab2159647. Latest consensus state: current_term: 1 leader_uuid: "f8e57311c3c4450eb776afcab2159647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e57311c3c4450eb776afcab2159647" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37643 } } }
I20260926 05:47:34.806368 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8e57311c3c4450eb776afcab2159647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8e57311c3c4450eb776afcab2159647" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37643 } } }
I20260926 05:47:34.806613 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [sys.catalog]: This master's current role is: LEADER
I20260926 05:47:34.806442 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647 [sys.catalog]: This master's current role is: LEADER
Sep 26 05:47:34 dist-test-slave-21p4 krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:34.807174 22533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:47:34.807758 22533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:47:34.810113 22533 catalog_manager.cc:1347] Generated new cluster ID: bad8f0907e38434786d2711f6d32c457
I20260926 05:47:34.810187 22533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:47:34.813073 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:34.799566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.11.247.62:37643)
0926 05:47:34.799803 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:34.799816 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:34.799855 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:47:34.800432 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:34.800440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:34.800537 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:34.800713 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:34.801426 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:34.802880 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:34.803704 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:34.804330 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:34.804456 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:34.807835 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:34.807841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:34.807841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:34.807869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:34.810094 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:34.810646 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:34.810651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:34.810651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:34.810781 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:34.811088 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:34.811090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:34.811090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:34.811511 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:34.811722 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:34.812262 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 05:47:34.812704 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":112,"threads_started":1}
I20260926 05:47:34.825984 22533 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:47:34.826864 22533 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:47:34.829895 22533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647: Generated new TSK 0
I20260926 05:47:34.830127 22533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:47:34.862546 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37643
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:34.955089 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:34.955368 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:34.955415 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:34.956794 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:34.956878 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:34.956930 22548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:47:34.958834 22548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:34.959151 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:34.959426 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:34.965154 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:34.965160 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:34.965158 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:34.965287 22548 server_base.cc:1034] running on GCE node
I20260926 05:47:34.965669 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:34.965976 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:34.967151 22548 hybrid_clock.cc:648] HybridClock initialized: now 1790401654967131 us; error 41 us; skew 500 ppm
Sep 26 05:47:34 dist-test-slave-21p4 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401654, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:34.970422 22548 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:47:34.971169 22548 webserver.cc:460] Webserver started at http://127.11.247.1:33037/ using document root <none> and password file <none>
I20260926 05:47:34.971444 22548 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:34.971499 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:34.971652 22548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:34.972934 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "bf2ee3d288644514817cad7f8a52fd19"
format_stamp: "Formatted at 2026-09-26 05:47:34 on dist-test-slave-21p4"
server_key: "ea8f22f75647b23ff57489202cfed583"
server_key_iv: "95dcbddf6da120bab0532ff3928f6af9"
server_key_version: "encryptionkey@0"
I20260926 05:47:34.973392 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2ee3d288644514817cad7f8a52fd19"
format_stamp: "Formatted at 2026-09-26 05:47:34 on dist-test-slave-21p4"
server_key: "ea8f22f75647b23ff57489202cfed583"
server_key_iv: "95dcbddf6da120bab0532ff3928f6af9"
server_key_version: "encryptionkey@0"
I20260926 05:47:34.976655 22548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 05:47:34.978075 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:34.978741 22548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:47:34.978879 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "bf2ee3d288644514817cad7f8a52fd19"
format_stamp: "Formatted at 2026-09-26 05:47:34 on dist-test-slave-21p4"
server_key: "ea8f22f75647b23ff57489202cfed583"
server_key_iv: "95dcbddf6da120bab0532ff3928f6af9"
server_key_version: "encryptionkey@0"
I20260926 05:47:34.979007 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:34.993324 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:34.996989 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:34.997236 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:34.997573 22548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:34.997941 22548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:34.998001 22548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:34.998042 22548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:34.998090 22548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.008198 22548 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:35553
I20260926 05:47:35.008306 22676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:35553 every 8 connection(s)
I20260926 05:47:35.008761 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:47:35.009325 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22548
I20260926 05:47:35.009438 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:47:35.009668 12252 external_mini_cluster.cc:1428] Setting key c0a508dd7c6d9815df5ea30a06d4ffa9
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401654, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
I20260926 05:47:35.021236 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.009772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:42727 (local address 127.11.247.62:37643)
0926 05:47:35.009845 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.009847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.010068 (+ 221us) server_negotiation.cc:407] Connection header received
0926 05:47:35.011034 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.011035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.011048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.011101 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:35.012397 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.012972 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.014169 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.014412 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.014506 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.017273 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:35.017275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:35.017276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:35.017285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:35.019329 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.019924 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.019926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.019926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.020004 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.020349 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.020351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.020351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.020485 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:35.020547 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.021002 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.021156 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:47:35.021278 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.009745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37643 (local address 127.11.247.1:42727)
0926 05:47:35.009922 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.009968 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.010913 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.011195 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.011200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.011823 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:35.012143 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.012150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.013133 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.013134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.014082 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.014085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.014523 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.014524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.014726 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.015350 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:35.015360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:35.017089 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:35.019452 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:35.019464 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:35.019467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:35.019822 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:35.020098 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:35.020099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:35.020099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:35.020232 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:35.020582 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:35.020589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:35.020759 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.020914 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.021070 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260926 05:47:35.022069 22677 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
I20260926 05:47:35.022210 22677 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:35.022439 22677 heartbeater.cc:511] Master 127.11.247.62:37643 requested a full tablet report, sending...
I20260926 05:47:35.023185 22486 ts_manager.cc:194] Registered new tserver with Master: bf2ee3d288644514817cad7f8a52fd19 (127.11.247.1:35553)
I20260926 05:47:35.024348 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:42727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:47:35.048956 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37643
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:35.140762 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:35.141108 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:35.141165 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:35.143810 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:35.143895 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:35.143934 22682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:47:35.145735 22682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:35.145993 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:35.146420 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:35.152446 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:35.152494 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:35.152511 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:35.152875 22682 server_base.cc:1034] running on GCE node
I20260926 05:47:35.153124 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:35.153484 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:35.154670 22682 hybrid_clock.cc:648] HybridClock initialized: now 1790401655154642 us; error 54 us; skew 500 ppm
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401655, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:35.157678 22682 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:47:35.158258 22682 webserver.cc:460] Webserver started at http://127.11.247.2:38597/ using document root <none> and password file <none>
I20260926 05:47:35.158537 22682 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:35.158646 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:35.158809 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:35.159992 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "36c5b0489ceb45c5975fb2c3a5521f26"
format_stamp: "Formatted at 2026-09-26 05:47:35 on dist-test-slave-21p4"
server_key: "c41415722c155007d29643b28721f5c7"
server_key_iv: "aee2e844e30b083c5140648beb3b603b"
server_key_version: "encryptionkey@0"
I20260926 05:47:35.160385 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "36c5b0489ceb45c5975fb2c3a5521f26"
format_stamp: "Formatted at 2026-09-26 05:47:35 on dist-test-slave-21p4"
server_key: "c41415722c155007d29643b28721f5c7"
server_key_iv: "aee2e844e30b083c5140648beb3b603b"
server_key_version: "encryptionkey@0"
I20260926 05:47:35.161819 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:47:35.162828 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.163126 22682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:35.163301 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "36c5b0489ceb45c5975fb2c3a5521f26"
format_stamp: "Formatted at 2026-09-26 05:47:35 on dist-test-slave-21p4"
server_key: "c41415722c155007d29643b28721f5c7"
server_key_iv: "aee2e844e30b083c5140648beb3b603b"
server_key_version: "encryptionkey@0"
I20260926 05:47:35.163426 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:35.174718 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:35.177258 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:35.177448 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:35.177721 22682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:35.178102 22682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:35.178159 22682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.178202 22682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:35.178232 22682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.184828 22682 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:44873
I20260926 05:47:35.184973 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:44873 every 8 connection(s)
I20260926 05:47:35.185272 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401655, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:35.195397 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22682
I20260926 05:47:35.195484 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:47:35.195716 12252 external_mini_cluster.cc:1428] Setting key ee3e3f58063f7a2df8bc6998ad0bdfed
I20260926 05:47:35.196210 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.186265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:46053 (local address 127.11.247.62:37643)
0926 05:47:35.186343 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.186347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.186623 (+ 276us) server_negotiation.cc:407] Connection header received
0926 05:47:35.187203 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.187204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.187215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.187270 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:35.188285 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.188818 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.189925 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.190150 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.190212 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.192537 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:35.192539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:35.192539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:35.192548 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:35.194120 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.194874 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.194876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.194876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.194961 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.195327 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.195328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.195328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.195527 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:35.195613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.196022 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.196131 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:47:35.196516 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.186254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37643 (local address 127.11.247.2:46053)
0926 05:47:35.186446 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.186514 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.187093 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.187329 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.187332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.187955 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:35.188174 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.188180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.188914 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.188915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.189832 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.189834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.190232 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.190233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.190339 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.190956 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:35.190966 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:35.192429 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:35.194242 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:35.194251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:35.194254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:35.194556 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:35.195083 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:35.195084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:35.195085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:35.195229 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:35.195600 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:35.195608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:35.195822 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.195957 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.196212 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260926 05:47:35.197083 22811 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37643
I20260926 05:47:35.197206 22811 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:35.197458 22811 heartbeater.cc:511] Master 127.11.247.62:37643 requested a full tablet report, sending...
I20260926 05:47:35.197919 22486 ts_manager.cc:194] Registered new tserver with Master: 36c5b0489ceb45c5975fb2c3a5521f26 (127.11.247.2:44873)
I20260926 05:47:35.198519 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:46053
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:47:35.227790 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37643
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:47:35.323452 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:47:35.323789 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:47:35.323855 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:47:35.325819 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:47:35.325935 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:47:35.326046 22816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:47:35.328053 22816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:47:35.328395 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:47:35.328668 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:47:35.334733 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:35.334715 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:47:35.334714 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:47:35.334733 22816 server_base.cc:1034] running on GCE node
I20260926 05:47:35.335238 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:47:35.335543 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:47:35.336707 22816 hybrid_clock.cc:648] HybridClock initialized: now 1790401655336683 us; error 40 us; skew 500 ppm
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401655, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:47:35.339879 22816 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:47:35.340394 22816 webserver.cc:460] Webserver started at http://127.11.247.3:38819/ using document root <none> and password file <none>
I20260926 05:47:35.340641 22816 fs_manager.cc:359] Metadata directory not provided
I20260926 05:47:35.340689 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:47:35.340830 22816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:47:35.341993 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "32b58f2b2be94732a409f0252cd48da6"
format_stamp: "Formatted at 2026-09-26 05:47:35 on dist-test-slave-21p4"
server_key: "22483f856bcd7664d84e20a3c27f53d0"
server_key_iv: "2fb43d5742fc3aec5446347c41e59996"
server_key_version: "encryptionkey@0"
I20260926 05:47:35.342408 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "32b58f2b2be94732a409f0252cd48da6"
format_stamp: "Formatted at 2026-09-26 05:47:35 on dist-test-slave-21p4"
server_key: "22483f856bcd7664d84e20a3c27f53d0"
server_key_iv: "2fb43d5742fc3aec5446347c41e59996"
server_key_version: "encryptionkey@0"
I20260926 05:47:35.343989 22816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:35.344985 22831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.345273 22816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:47:35.345417 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "32b58f2b2be94732a409f0252cd48da6"
format_stamp: "Formatted at 2026-09-26 05:47:35 on dist-test-slave-21p4"
server_key: "22483f856bcd7664d84e20a3c27f53d0"
server_key_iv: "2fb43d5742fc3aec5446347c41e59996"
server_key_version: "encryptionkey@0"
I20260926 05:47:35.345495 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:47:35.361181 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:47:35.363674 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:47:35.363839 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:47:35.364317 22816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:47:35.364755 22816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:47:35.364794 22816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.364818 22816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:47:35.364871 22816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:47:35.371647 22816 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:42989
I20260926 05:47:35.371733 22944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:42989 every 8 connection(s)
I20260926 05:47:35.372197 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:47:35.374832 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 22816
I20260926 05:47:35.374918 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:47:35.375130 12252 external_mini_cluster.cc:1428] Setting key 086215af41e75c4ef2640a89e85579fa
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401655, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:47:35.383574 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.373185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:58799 (local address 127.11.247.62:37643)
0926 05:47:35.373255 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.373257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.373389 (+ 132us) server_negotiation.cc:407] Connection header received
0926 05:47:35.373878 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.373879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.373892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.373960 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:35.375146 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.375923 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.376987 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.377217 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.377302 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.380280 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:35.380282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:35.380283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:35.380293 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:35.381817 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.382347 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.382349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.382349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.382421 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.382775 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.382776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.382776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.382917 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:35.383024 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.383369 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.383463 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:47:35.383880 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.373027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37643 (local address 127.11.247.3:58799)
0926 05:47:35.373202 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.373247 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.373753 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.374057 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.374062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.374742 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:47:35.375021 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.375035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.376024 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.376025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.376881 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.376885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.377316 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.377327 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.377491 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.378084 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:47:35.378095 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:47:35.380138 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:47:35.381934 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:35.381947 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:35.381951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:35.382256 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:35.382500 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:47:35.382501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:47:35.382501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:47:35.382703 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:47:35.383016 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:47:35.383024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:47:35.383215 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.383329 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.383547 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260926 05:47:35.385026 22945 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37643
I20260926 05:47:35.385138 22945 heartbeater.cc:464] Registering TS with master...
I20260926 05:47:35.385380 22945 heartbeater.cc:511] Master 127.11.247.62:37643 requested a full tablet report, sending...
I20260926 05:47:35.386106 22486 ts_manager.cc:194] Registered new tserver with Master: 32b58f2b2be94732a409f0252cd48da6 (127.11.247.3:42989)
I20260926 05:47:35.386634 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:58799
I20260926 05:47:35.387493 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:47:35.396927 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.388931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.11.247.62:37643)
0926 05:47:35.389013 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.389016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.389147 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:47:35.389254 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.389255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.389269 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.389332 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:47:35.390356 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.391007 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.391791 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.391994 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.392085 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.393419 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:47:35.393422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:47:35.393423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:47:35.393445 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:47:35.395184 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.395807 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.395809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.395809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.395881 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:47:35.396190 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:47:35.396191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:47:35.396191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:47:35.396386 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:47:35.396520 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.396704 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.396813 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 05:47:35.400027 22486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:47:35.402071 22486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:47:35.410862 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.405710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.11.247.2:44873)
0926 05:47:35.405924 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.405929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.406063 (+ 134us) server_negotiation.cc:407] Connection header received
0926 05:47:35.406215 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.406218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.406240 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.406295 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:35.406564 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.407874 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.409723 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.410494 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.410562 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.410640 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.410720 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.410777 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260926 05:47:35.410849 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.405579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:44873 (local address 127.0.0.1:33086)
0926 05:47:35.405976 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.405985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.406075 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.406426 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.406427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.406429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:35.406476 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.406478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.408138 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.408139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.409418 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.409421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.410561 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.410562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.410651 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.410655 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.410706 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.410746 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":71,"thread_start_us":93,"threads_started":1}
I20260926 05:47:35.411038 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.405585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.11.247.1:35553)
0926 05:47:35.405792 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.405799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.406100 (+ 301us) server_negotiation.cc:407] Connection header received
0926 05:47:35.406167 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.406170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.406194 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.406334 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:35.406626 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.407857 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.409329 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.410382 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.410487 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.410551 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.410894 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.410948 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260926 05:47:35.411247 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.406056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.11.247.3:42989)
0926 05:47:35.406287 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.406292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.406301 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:35.406336 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.406338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.406436 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.406551 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:35.406791 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.407728 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.409755 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.410315 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.410415 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.410493 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.411109 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.411170 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260926 05:47:35.411428 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.405763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:42989 (local address 127.0.0.1:48758)
0926 05:47:35.406191 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.406200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.406242 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.406659 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.406660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.406663 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:35.406712 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.406714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.407850 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.407851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.409646 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.409648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.410441 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.410443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.410548 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.410566 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.410661 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.410740 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":40,"thread_start_us":81,"threads_started":1}
I20260926 05:47:35.411511 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.405509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35553 (local address 127.0.0.1:54494)
0926 05:47:35.405924 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.405982 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.406075 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.406363 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.406366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.406368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:35.406505 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.406507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.407988 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.407989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.409200 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.409204 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.410638 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.410639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.410720 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.410726 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.410779 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.410826 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":126,"thread_start_us":84,"threads_started":1}
I20260926 05:47:35.412905 22745 tablet_service.cc:1467] Processing CreateTablet for tablet d3a2dd84b2f744d89de4dd146afbd429 (DEFAULT_TABLE table=test-table [id=6ea40eb9e8054ccabfc379005c4628de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:35.413314 22745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a2dd84b2f744d89de4dd146afbd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:35.413156 22879 tablet_service.cc:1467] Processing CreateTablet for tablet d3a2dd84b2f744d89de4dd146afbd429 (DEFAULT_TABLE table=test-table [id=6ea40eb9e8054ccabfc379005c4628de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:35.413148 22611 tablet_service.cc:1467] Processing CreateTablet for tablet d3a2dd84b2f744d89de4dd146afbd429 (DEFAULT_TABLE table=test-table [id=6ea40eb9e8054ccabfc379005c4628de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:47:35.413621 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a2dd84b2f744d89de4dd146afbd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:35.413658 22879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a2dd84b2f744d89de4dd146afbd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:47:35.416843 22962 tablet_bootstrap.cc:492] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: Bootstrap starting.
I20260926 05:47:35.417689 22962 tablet_bootstrap.cc:654] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:35.418213 22962 log.cc:824] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:35.418511 22963 tablet_bootstrap.cc:492] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Bootstrap starting.
I20260926 05:47:35.418738 22964 tablet_bootstrap.cc:492] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: Bootstrap starting.
I20260926 05:47:35.419378 22964 tablet_bootstrap.cc:654] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:35.419378 22962 tablet_bootstrap.cc:492] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: No bootstrap required, opened a new log
I20260926 05:47:35.419413 22963 tablet_bootstrap.cc:654] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Neither blocks nor log segments found. Creating new log.
I20260926 05:47:35.419515 22962 ts_tablet_manager.cc:1397] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:47:35.419951 22964 log.cc:824] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:35.420022 22963 log.cc:824] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:47:35.420765 22964 tablet_bootstrap.cc:492] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: No bootstrap required, opened a new log
I20260926 05:47:35.420778 22963 tablet_bootstrap.cc:492] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: No bootstrap required, opened a new log
I20260926 05:47:35.420838 22964 ts_tablet_manager.cc:1397] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:35.420838 22963 ts_tablet_manager.cc:1397] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:47:35.421861 22962 raft_consensus.cc:348] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.422081 22962 raft_consensus.cc:374] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:35.422111 22962 raft_consensus.cc:729] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c5b0489ceb45c5975fb2c3a5521f26, State: Initialized, Role: FOLLOWER
I20260926 05:47:35.422242 22962 consensus_queue.cc:260] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.422535 22811 heartbeater.cc:503] Master 127.11.247.62:37643 was elected leader, sending a full tablet report...
I20260926 05:47:35.422554 22962 ts_tablet_manager.cc:1428] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:47:35.423419 22964 raft_consensus.cc:348] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.423419 22963 raft_consensus.cc:348] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.423600 22964 raft_consensus.cc:374] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:35.423600 22963 raft_consensus.cc:374] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:47:35.423638 22964 raft_consensus.cc:729] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ee3d288644514817cad7f8a52fd19, State: Initialized, Role: FOLLOWER
I20260926 05:47:35.423638 22963 raft_consensus.cc:729] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b58f2b2be94732a409f0252cd48da6, State: Initialized, Role: FOLLOWER
I20260926 05:47:35.423772 22963 consensus_queue.cc:260] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.423804 22964 consensus_queue.cc:260] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.424136 22963 ts_tablet_manager.cc:1428] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260926 05:47:35.424136 22964 ts_tablet_manager.cc:1428] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:47:35.424161 22945 heartbeater.cc:503] Master 127.11.247.62:37643 was elected leader, sending a full tablet report...
I20260926 05:47:35.424283 22677 heartbeater.cc:503] Master 127.11.247.62:37643 was elected leader, sending a full tablet report...
W20260926 05:47:35.435900 22812 tablet.cc:2369] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:47:35.466737 22970 raft_consensus.cc:482] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:47:35.466917 22970 raft_consensus.cc:504] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.467489 22970 leader_election.cc:288] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2ee3d288644514817cad7f8a52fd19 (127.11.247.1:35553), 36c5b0489ceb45c5975fb2c3a5521f26 (127.11.247.2:44873)
I20260926 05:47:35.472087 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.467777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:41813 (local address 127.11.247.2:44873)
0926 05:47:35.467829 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.467832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.467843 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:47:35.467902 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.467903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.467914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.468072 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:35.468319 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.469206 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.471034 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.471636 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.471725 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.471734 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.471949 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.472026 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:47:35.472087 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.467765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:53437 (local address 127.11.247.1:35553)
0926 05:47:35.467814 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.467816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.468291 (+ 475us) server_negotiation.cc:407] Connection header received
0926 05:47:35.468382 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.468383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.468395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.468456 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:47:35.468611 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.469288 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.470995 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.471634 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.471725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.471734 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.471949 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.471999 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:47:35.472149 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.467708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:44873 (local address 127.11.247.3:41813)
0926 05:47:35.467762 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.467775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.467829 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.468109 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.468111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.468113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:35.468199 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.468201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.469323 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.469324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.470853 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.470858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.471760 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.471762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.471856 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.471858 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.471921 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.472027 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":9}
I20260926 05:47:35.472163 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.467709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35553 (local address 127.11.247.3:53437)
0926 05:47:35.468101 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.468111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.468195 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.468473 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:47:35.468474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:47:35.468476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:47:35.468531 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.468532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.469388 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.469389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.470853 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:47:35.470858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.471760 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.471762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:47:35.471851 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.471856 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:47:35.471920 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:47:35.472019 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":228,"thread_start_us":144,"threads_started":1}
I20260926 05:47:35.472695 22765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a2dd84b2f744d89de4dd146afbd429" candidate_uuid: "32b58f2b2be94732a409f0252cd48da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" is_pre_election: true
I20260926 05:47:35.472887 22765 raft_consensus.cc:2454] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32b58f2b2be94732a409f0252cd48da6 in term 0.
I20260926 05:47:35.473260 22833 leader_election.cc:302] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b58f2b2be94732a409f0252cd48da6, 36c5b0489ceb45c5975fb2c3a5521f26; no voters:
I20260926 05:47:35.473193 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a2dd84b2f744d89de4dd146afbd429" candidate_uuid: "32b58f2b2be94732a409f0252cd48da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ee3d288644514817cad7f8a52fd19" is_pre_election: true
I20260926 05:47:35.473484 22970 raft_consensus.cc:2781] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:47:35.473554 22970 raft_consensus.cc:482] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:47:35.473558 22631 raft_consensus.cc:2454] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32b58f2b2be94732a409f0252cd48da6 in term 0.
I20260926 05:47:35.473585 22970 raft_consensus.cc:3037] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:35.474411 22970 raft_consensus.cc:504] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.474649 22970 leader_election.cc:288] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [CANDIDATE]: Term 1 election: Requested vote from peers bf2ee3d288644514817cad7f8a52fd19 (127.11.247.1:35553), 36c5b0489ceb45c5975fb2c3a5521f26 (127.11.247.2:44873)
I20260926 05:47:35.474892 22765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a2dd84b2f744d89de4dd146afbd429" candidate_uuid: "32b58f2b2be94732a409f0252cd48da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36c5b0489ceb45c5975fb2c3a5521f26"
I20260926 05:47:35.474895 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a2dd84b2f744d89de4dd146afbd429" candidate_uuid: "32b58f2b2be94732a409f0252cd48da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2ee3d288644514817cad7f8a52fd19"
I20260926 05:47:35.474980 22765 raft_consensus.cc:3037] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:35.475008 22631 raft_consensus.cc:3037] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:47:35.475747 22631 raft_consensus.cc:2454] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b58f2b2be94732a409f0252cd48da6 in term 1.
I20260926 05:47:35.475893 22765 raft_consensus.cc:2454] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b58f2b2be94732a409f0252cd48da6 in term 1.
I20260926 05:47:35.476171 22835 leader_election.cc:302] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b58f2b2be94732a409f0252cd48da6, bf2ee3d288644514817cad7f8a52fd19; no voters:
I20260926 05:47:35.476318 22970 raft_consensus.cc:2781] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:47:35.476615 22970 raft_consensus.cc:686] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 1 LEADER]: Becoming Leader. State: Replica: 32b58f2b2be94732a409f0252cd48da6, State: Running, Role: LEADER
I20260926 05:47:35.476852 22970 consensus_queue.cc:237] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } }
I20260926 05:47:35.477945 22485 catalog_manager.cc:5495] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32b58f2b2be94732a409f0252cd48da6 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "32b58f2b2be94732a409f0252cd48da6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32b58f2b2be94732a409f0252cd48da6" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20260926 05:47:35.487016 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.483595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.11.247.3:42989)
0926 05:47:35.483636 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:47:35.483638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:47:35.483647 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:47:35.483713 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:47:35.483714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:47:35.483730 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:47:35.483796 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:47:35.484230 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.484733 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.485837 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:47:35.486038 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:47:35.486104 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:47:35.486501 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:47:35.486702 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:47:35.486849 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 05:47:35.486899 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 05:47:35.486924 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:47:35.489553 22631 raft_consensus.cc:1261] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19 [term 1 FOLLOWER]: Refusing update from remote peer 32b58f2b2be94732a409f0252cd48da6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:35.489564 22765 raft_consensus.cc:1261] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 1 FOLLOWER]: Refusing update from remote peer 32b58f2b2be94732a409f0252cd48da6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:47:35.489992 22970 consensus_queue.cc:1035] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36c5b0489ceb45c5975fb2c3a5521f26" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:35.490365 22970 consensus_queue.cc:1035] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2ee3d288644514817cad7f8a52fd19" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:47:35.492982 22975 mvcc.cc:205] Tried to move back new op lower bound from 7333485180881879040 to 7333485180834082816. Current Snapshot: MvccSnapshot[applied={T|T < 7333485180881879040}]
I20260926 05:47:35.496132 22486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:35.496280 22486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:47:35.497061 22486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8e57311c3c4450eb776afcab2159647: Sending DeleteTablet for 3 replicas of tablet d3a2dd84b2f744d89de4dd146afbd429
I20260926 05:47:35.497459 22611 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a2dd84b2f744d89de4dd146afbd429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:35 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:54494
I20260926 05:47:35.497483 22879 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a2dd84b2f744d89de4dd146afbd429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:35 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:48758
I20260926 05:47:35.497505 22745 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a2dd84b2f744d89de4dd146afbd429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:47:35 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:33086
I20260926 05:47:35.497884 22987 tablet_replica.cc:330] T d3a2dd84b2f744d89de4dd146afbd429 P bf2ee3d288644514817cad7f8a52fd19: stopping tablet replica
I20260926 05:47:35.497964 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22548
W20260926 05:47:35.504823 22473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:35553 (error 108)
I20260926 05:47:35.505254 22986 tablet_replica.cc:330] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26: stopping tablet replica
I20260926 05:47:35.505441 22986 raft_consensus.cc:2229] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:47:35.505605 22986 raft_consensus.cc:2258] T d3a2dd84b2f744d89de4dd146afbd429 P 36c5b0489ceb45c5975fb2c3a5521f26 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 05:47:35.505656 22960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:47:35.505340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35553 (local address 127.0.0.1:54504)
0926 05:47:35.505400 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.505411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:47:35.505464 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:47:35.505590 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:35553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:47:35.505789 22473 catalog_manager.cc:4584] TS bf2ee3d288644514817cad7f8a52fd19 (127.11.247.1:35553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3a2dd84b2f744d89de4dd146afbd429: Network error: Client connection negotiation failed: client connection to 127.11.247.1:35553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:47:35.505815 22985 tablet_replica.cc:330] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: stopping tablet replica
I20260926 05:47:35.505935 22985 raft_consensus.cc:2229] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:47:35.506060 22985 raft_consensus.cc:2258] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:47:35.506301 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22682
I20260926 05:47:35.506775 22985 ts_tablet_manager.cc:1905] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:47:35.509397 22985 ts_tablet_manager.cc:1918] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:47:35.509577 22985 log.cc:1196] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/d3a2dd84b2f744d89de4dd146afbd429
I20260926 05:47:35.509876 22985 ts_tablet_manager.cc:1939] T d3a2dd84b2f744d89de4dd146afbd429 P 32b58f2b2be94732a409f0252cd48da6: Deleting consensus metadata
I20260926 05:47:35.510421 22473 catalog_manager.cc:4841] TS 32b58f2b2be94732a409f0252cd48da6 (127.11.247.3:42989): tablet d3a2dd84b2f744d89de4dd146afbd429 (table test-table [id=6ea40eb9e8054ccabfc379005c4628de]) successfully deleted
I20260926 05:47:35.514114 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:47:35.513921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:44873 (local address 127.0.0.1:33092)
0926 05:47:35.513993 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:47:35.514070 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.2:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 05:47:35.514299 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22816
I20260926 05:47:35.520263 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 22454
2026-09-26T05:47:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2109 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22991](info): set up 2 sockets
Sep 26 05:47:35 dist-test-slave-21p4 krb5kdc[22991](info): commencing operation
krb5kdc: starting...
W20260926 05:47:36.616688 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:47:36 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:47:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/rangeradmin_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/rangeradmin_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/rangerlookup_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/rangerlookup_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
I20260926 05:47:36.823464 12252 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksryJ5D/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/postgres -l logfile start
2026-09-26 05:47:40.099 UTC [23021] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 05:47:40.099 UTC [23021] LOG: listening on IPv4 address "127.11.247.20", port 51441
2026-09-26 05:47:40.106 UTC [23021] LOG: listening on Unix socket "/tmp/.s.PGSQL.51441"
2026-09-26 05:47:40.131 UTC [23024] LOG: database system was shut down at 2026-09-26 05:47:38 UTC
2026-09-26 05:47:40.135 UTC [23021] LOG: database system is ready to accept connections
I20260926 05:47:41.103627 12252 mini_postgres.cc:96] Postgres bound to 51441
2026-09-26 05:47:41.109 UTC [23032] FATAL: database "slave" does not exist
127.11.247.20:51441 - accepting connections
I20260926 05:47:41.110728 12252 mini_ranger.cc:161] Starting Ranger...
I20260926 05:47:41.126919 12252 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 05:47:41.740447 12252 mini_ranger.cc:87] Created ranger Postgres database
I20260926 05:47:41.740543 12252 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-admin
2026-09-26 05:47:42,100 [I] DB FLAVOR :POSTGRES
2026-09-26 05:47:42,100 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:47:42,100 [I] Checking connection..
2026-09-26 05:47:42,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 05:47:42,357 [I] Checking connection passed.
2026-09-26 05:47:42,358 [I] --------- Verifying version history table ---------
2026-09-26 05:47:42,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:47:42,616 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:47:42,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:47:42,866 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:47:42,866 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 05:47:42,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 05:47:43.123 UTC [23096] WARNING: there is no transaction in progress
2026-09-26 05:47:43,130 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 05:47:43,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:47:43,403 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 05:47:43,403 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:47:43,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 05:47:43,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 05:47:43,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-21p4.c.gcp-upstream.internal','N') ;"
2026-09-26 05:47:44,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:47:44,443 [I] Table x_portal_user does not exist in database ranger
2026-09-26 05:47:44,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:47:44,706 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 05:47:44,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:47:44,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:47:45,225 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 05:47:45,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 05:47:46.410 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.431 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.452 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.473 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.494 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.515 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.584 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.594 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.613 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.626 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.641 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.657 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.670 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.680 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.691 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:46.700 UTC [23204] WARNING: there is no transaction in progress
2026-09-26 05:47:47,634 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 05:47:47,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 05:47:47,894 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 05:47:47,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:47:48,164 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 05:47:48,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:47:48,438 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 05:47:48,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:47:48,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:47:49,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-21p4.c.gcp-upstream.internal';"
2026-09-26 05:47:49,267 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 05:47:49,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 05:47:49,525 [I] DB_PATCHES have already been applied
I20260926 05:47:49.531023 12252 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksryJ5D/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 05:47:49.531123 12252 mini_ranger.cc:193] Using host: 127.11.247.20
I20260926 05:47:49.534538 12252 mini_ranger.cc:227] Ranger admin URL: http://127.11.247.20:45655
Sep 26, 2026 5:47:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:47:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 5:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 5:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 5:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 5:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 05:47:50.697688 12252 mini_ranger.cc:160] Time spent starting Ranger: real 9.587s user 0.000s sys 0.010s
Sep 26, 2026 5:48:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 5:48:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 5:48:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 5:48:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 5:48:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 05:48:14.221424 12252 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/rangerkms_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/rangerkms_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.247.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.247.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 05:48:14 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401694, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 05:48:14.314080 12252 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 05:48:14.324810 12252 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 05:48:15.479671 12252 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 05:48:15.479768 12252 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms
I20260926 05:48:15.479787 12252 mini_ranger_kms.cc:227] Using postgres at 127.11.247.20:51441
2026-09-26 05:48:15,838 [I] DB FLAVOR :POSTGRES
2026-09-26 05:48:15,839 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:48:15,839 [I] Checking connection
2026-09-26 05:48:15,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 05:48:16,123 [I] connection success
2026-09-26 05:48:16,123 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 05:48:16,123 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 05:48:16,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 05:48:16,396 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 05:48:16,396 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:48:16,397 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 05:48:16,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 05:48:16,673 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 05:48:17.223374 12252 mini_ranger_kms.cc:310] Created kms service
I20260926 05:48:17.436600 12252 mini_ranger_kms.cc:326] Created kudu user
I20260926 05:48:17.481660 12252 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 05:48:17 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401694, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:48:17.926309 12252 mini_ranger_kms.cc:384] Added ranger policy
I20260926 05:48:17.926435 12252 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 05:48:17.926458 12252 mini_ranger_kms.cc:240] Using host: 127.11.247.20
I20260926 05:48:17.930533 12252 mini_ranger_kms.cc:276] Ranger KMS PID: 23385
I20260926 05:48:17.930721 12252 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.247.20:33555
Sep 26, 2026 5:48:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:48:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:48:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 5:48:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 5:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 5:48:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 05:48:20 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.247.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:48:20.227795 12252 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.914s user 0.008s sys 0.208s
I20260926 05:48:20.227933 12252 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 05:48:20.228029 12252 mini_ranger_kms.cc:401] 127.11.247.20:33555/kms/v1/keys
Sep 26 05:48:20 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.247.20@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
Sep 26 05:48:20 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.247.20@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
Sep 26 05:48:23 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:48:23.302951 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:43797
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:43797
--ranger_config_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:33555/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:48:23.390791 23435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:48:23.391027 23435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:48:23.391052 23435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:48:23.392310 23435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:48:23.392360 23435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:48:23.392374 23435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:48:23.392386 23435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:48:23.392400 23435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:48:23.394059 23435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.247.20:33555/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:43797
--ranger_config_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:48:23.394518 23435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:48:23.395056 23435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:48:23.401794 23441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:48:23.401806 23443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:48:23.401841 23435 server_base.cc:1034] running on GCE node
W20260926 05:48:23.401827 23440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:48:23.402499 23435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:48:23.402877 23435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:48:23.404085 23435 hybrid_clock.cc:648] HybridClock initialized: now 1790401703404058 us; error 41 us; skew 500 ppm
Sep 26 05:48:23 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401703, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:48:23.411474 23435 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:48:23.412042 23435 webserver.cc:460] Webserver started at http://127.11.247.62:41871/ using document root <none> and password file <none>
I20260926 05:48:23.412350 23435 fs_manager.cc:359] Metadata directory not provided
I20260926 05:48:23.412420 23435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:48:23.412588 23435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:48:23 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401703, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:48:23.471197 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "d3fdfa3cffc3451ba768b785c1b1d040"
format_stamp: "Formatted at 2026-09-26 05:48:23 on dist-test-slave-21p4"
server_key: "34058c7e451741645599386bc910a66b"
server_key_iv: "8d45531fc6fbb00cf2c6335147f3a676"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:23.471755 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "d3fdfa3cffc3451ba768b785c1b1d040"
format_stamp: "Formatted at 2026-09-26 05:48:23 on dist-test-slave-21p4"
server_key: "34058c7e451741645599386bc910a66b"
server_key_iv: "8d45531fc6fbb00cf2c6335147f3a676"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:23.473703 23435 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:48:23.489516 23450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:23.489796 23435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:48:23.489931 23435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "d3fdfa3cffc3451ba768b785c1b1d040"
format_stamp: "Formatted at 2026-09-26 05:48:23 on dist-test-slave-21p4"
server_key: "34058c7e451741645599386bc910a66b"
server_key_iv: "8d45531fc6fbb00cf2c6335147f3a676"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:23.490051 23435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:48:23.501986 23435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:48:23.504459 23435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:48:23.504630 23435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:48:23.509905 23435 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:43797
I20260926 05:48:23.509932 23502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:43797 every 8 connection(s)
I20260926 05:48:23.510353 23435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:48:23.511061 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:48:23.513495 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040: Bootstrap starting.
I20260926 05:48:23.514144 23503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040: Neither blocks nor log segments found. Creating new log.
I20260926 05:48:23.514456 23503 log.cc:824] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040: Log is configured to *not* fsync() on all Append() calls
I20260926 05:48:23.515228 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040: No bootstrap required, opened a new log
I20260926 05:48:23.517549 23503 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43797 } }
I20260926 05:48:23.517716 23503 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:48:23.517740 23503 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fdfa3cffc3451ba768b785c1b1d040, State: Initialized, Role: FOLLOWER
I20260926 05:48:23.517904 23503 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43797 } }
I20260926 05:48:23.517983 23503 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:48:23.518024 23503 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:48:23.518074 23503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:48:23.518761 23503 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43797 } }
I20260926 05:48:23.518860 23503 leader_election.cc:302] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3fdfa3cffc3451ba768b785c1b1d040; no voters:
I20260926 05:48:23.519106 23503 leader_election.cc:288] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:48:23.519135 23506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:48:23.519269 23506 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [term 1 LEADER]: Becoming Leader. State: Replica: d3fdfa3cffc3451ba768b785c1b1d040, State: Running, Role: LEADER
I20260926 05:48:23.519390 23503 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:48:23.519430 23506 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43797 } }
I20260926 05:48:23.519708 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 23435
I20260926 05:48:23.519824 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:48:23.519835 23506 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3fdfa3cffc3451ba768b785c1b1d040. Latest consensus state: current_term: 1 leader_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43797 } } }
I20260926 05:48:23.519901 23506 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [sys.catalog]: This master's current role is: LEADER
I20260926 05:48:23.519850 23508 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3fdfa3cffc3451ba768b785c1b1d040" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43797 } } }
I20260926 05:48:23.519943 23508 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040 [sys.catalog]: This master's current role is: LEADER
I20260926 05:48:23.520044 12252 external_mini_cluster.cc:1428] Setting key 1e2fa6546f3d6b4e7fb31241e33a8c41
I20260926 05:48:23.521870 23503 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:48:23.524627 23510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:48:23.526103 23510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:48:23 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:48:23.537345 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:23.520748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.11.247.62:43797)
0926 05:48:23.520947 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:23.520958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:23.521054 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:48:23.521195 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:23.521203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:23.521264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:23.521453 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:48:23.524173 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:23.526452 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:23.527649 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:23.528427 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:23.531896 (+ 3469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:23.532024 (+ 128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:48:23.532028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:48:23.532028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:48:23.532038 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:48:23.535090 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:23.535573 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:23.535577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:23.535577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:23.535701 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:23.535943 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:23.535944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:23.535944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:23.536301 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:48:23.536883 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:23.536961 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:48:23.537094 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260926 05:48:23.539149 23510 catalog_manager.cc:1347] Generated new cluster ID: 3d6d75cacf284441835177d88b950d0b
I20260926 05:48:23.539281 23510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:48:23.578553 23510 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:48:23.579349 23510 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:48:23.590797 23510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040: Generated new TSK 0
I20260926 05:48:23.591138 23510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:48:24.274513 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:33555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:43797
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:48:24.363085 23548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:48:24.363332 23548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:48:24.363370 23548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:48:24.364590 23548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:48:24.364662 23548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:48:24.364713 23548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:48:24.366269 23548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.247.20:33555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:48:24.366550 23548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:48:24.366870 23548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:48:24.372205 23556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:48:24.372172 23554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:48:24.372197 23553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:48:24.372763 23548 server_base.cc:1034] running on GCE node
I20260926 05:48:24.373029 23548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:48:24.373356 23548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:48:24.374565 23548 hybrid_clock.cc:648] HybridClock initialized: now 1790401704374541 us; error 36 us; skew 500 ppm
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:48:24.378000 23548 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:48:24.378633 23548 webserver.cc:460] Webserver started at http://127.11.247.1:34217/ using document root <none> and password file <none>
I20260926 05:48:24.378963 23548 fs_manager.cc:359] Metadata directory not provided
I20260926 05:48:24.379041 23548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:48:24.379194 23548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:48:24.392629 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "b14960e65a17465db4484efac42c97a3"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "458c1e968e559d73cbf4bd9d31b5ec97"
server_key_iv: "ae8c6b3999d9735c2114bdb31c0ad066"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.393210 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "b14960e65a17465db4484efac42c97a3"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "458c1e968e559d73cbf4bd9d31b5ec97"
server_key_iv: "ae8c6b3999d9735c2114bdb31c0ad066"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.395056 23548 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 05:48:24.410776 23563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.411001 23548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 05:48:24.411095 23548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "b14960e65a17465db4484efac42c97a3"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "458c1e968e559d73cbf4bd9d31b5ec97"
server_key_iv: "ae8c6b3999d9735c2114bdb31c0ad066"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.411211 23548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:48:24.435321 23548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:48:24.437614 23548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:48:24.437795 23548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:48:24.438072 23548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:48:24.438457 23548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:48:24.438508 23548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.438602 23548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:48:24.438644 23548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.447506 23548 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:44721
I20260926 05:48:24.448061 23548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:48:24.448272 23677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:44721 every 8 connection(s)
I20260926 05:48:24.454181 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 23548
I20260926 05:48:24.454303 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:48:24.454509 12252 external_mini_cluster.cc:1428] Setting key 6fa634bca47fb759e1de97b71b9fc6bd
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:48:24.463748 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:24.448894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:43797 (local address 127.11.247.1:53243)
0926 05:48:24.449085 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:24.449130 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:24.449839 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:24.452941 (+ 3102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:24.452947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:24.453255 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:48:24.453578 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:24.453585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.454753 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.454754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:24.456258 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:24.456262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.456856 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.456858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:24.457059 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:24.457771 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:48:24.457782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:48:24.459549 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:48:24.461785 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:48:24.461797 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:48:24.461800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:48:24.462176 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:48:24.462467 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:48:24.462468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:48:24.462468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:48:24.462654 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:48:24.463006 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:48:24.463014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:48:24.463250 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 05:48:24.463382 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 05:48:24.463554 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260926 05:48:24.464200 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:24.448933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:53243 (local address 127.11.247.62:43797)
0926 05:48:24.452629 (+ 3696us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:24.452632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:24.452646 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:48:24.452677 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:24.452678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:24.452693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:24.452793 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:48:24.453735 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.454574 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.456382 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.456709 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.456779 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:24.459702 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:48:24.459704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:48:24.459704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:48:24.459714 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:48:24.461657 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:24.462295 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:24.462296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:24.462297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:24.462377 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:24.462740 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:24.462741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:24.462741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:24.462903 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:48:24.462975 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:24.463942 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 05:48:24.464053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3658,"thread_start_us":100,"threads_started":1}
I20260926 05:48:24.464612 23678 heartbeater.cc:347] Connected to a master server at 127.11.247.62:43797
I20260926 05:48:24.464725 23678 heartbeater.cc:464] Registering TS with master...
I20260926 05:48:24.464948 23678 heartbeater.cc:511] Master 127.11.247.62:43797 requested a full tablet report, sending...
I20260926 05:48:24.465552 23467 ts_manager.cc:194] Registered new tserver with Master: b14960e65a17465db4484efac42c97a3 (127.11.247.1:44721)
I20260926 05:48:24.466557 23467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:53243
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:48:24.494268 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:33555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:43797
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:48:24.611732 23685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:48:24.612156 23685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:48:24.612222 23685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:48:24.614223 23685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:48:24.614341 23685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:48:24.614408 23685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:48:24.617324 23685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.247.20:33555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:48:24.617663 23685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:48:24.618069 23685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:48:24.625277 23695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:48:24.625330 23685 server_base.cc:1034] running on GCE node
W20260926 05:48:24.625458 23693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:48:24.625785 23692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:48:24.626089 23685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:48:24.626421 23685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:48:24.627583 23685 hybrid_clock.cc:648] HybridClock initialized: now 1790401704627556 us; error 38 us; skew 500 ppm
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:48:24.631131 23685 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:48:24.631798 23685 webserver.cc:460] Webserver started at http://127.11.247.2:44093/ using document root <none> and password file <none>
I20260926 05:48:24.632201 23685 fs_manager.cc:359] Metadata directory not provided
I20260926 05:48:24.632274 23685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:48:24.632469 23685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:48:24.647322 23685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "e6a6dea3e54f433fbf07b283f9fcac9d"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "5403fa7a3461b12d2b23b2b4032fa5c1"
server_key_iv: "727a12ffcd725487d67d39cdc69d5ac3"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.647816 23685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "e6a6dea3e54f433fbf07b283f9fcac9d"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "5403fa7a3461b12d2b23b2b4032fa5c1"
server_key_iv: "727a12ffcd725487d67d39cdc69d5ac3"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.662289 23685 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.002s sys 0.000s
I20260926 05:48:24.683477 23702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.683749 23685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:48:24.683836 23685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "e6a6dea3e54f433fbf07b283f9fcac9d"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "5403fa7a3461b12d2b23b2b4032fa5c1"
server_key_iv: "727a12ffcd725487d67d39cdc69d5ac3"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.683946 23685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:48:24.699631 23685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:48:24.702580 23685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:48:24.702874 23685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:48:24.703181 23685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:48:24.703681 23685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:48:24.703778 23685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.704149 23685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:48:24.704221 23685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.713323 23685 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:43453
I20260926 05:48:24.713974 23685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:48:24.717283 23816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:43453 every 8 connection(s)
I20260926 05:48:24.717396 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 23685
I20260926 05:48:24.717509 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:48:24.717742 12252 external_mini_cluster.cc:1428] Setting key 7e29d0501e4b9b070109989e29058feb
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:48:24.731072 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:24.716170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:57101 (local address 127.11.247.62:43797)
0926 05:48:24.716646 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:24.716648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:24.719988 (+ 3340us) server_negotiation.cc:407] Connection header received
0926 05:48:24.720808 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:24.720810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:24.720828 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:24.720925 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:48:24.721994 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.722799 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.724342 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.724676 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.724726 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:24.727569 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:48:24.727571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:48:24.727571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:48:24.727580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:48:24.729285 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:24.729841 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:24.729842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:24.729842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:24.729907 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:24.730202 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:24.730204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:24.730204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:24.730351 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:48:24.730418 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:24.730869 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 05:48:24.730971 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260926 05:48:24.732014 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:24.715180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:43797 (local address 127.11.247.2:57101)
0926 05:48:24.719799 (+ 4619us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:24.719899 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:24.720629 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:24.721214 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:24.721220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:24.721505 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:48:24.721886 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:24.721893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.722957 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.722958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:24.724230 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:24.724234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:24.724838 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:24.724839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:24.724997 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:24.725644 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:48:24.725656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:48:24.727440 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:48:24.729396 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:48:24.729405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:48:24.729408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:48:24.729744 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:48:24.729984 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:48:24.729985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:48:24.729985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:48:24.730127 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:48:24.730474 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:48:24.730480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:48:24.730701 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 05:48:24.731577 (+ 876us) client_negotiation.cc:240] Negotiation successful
0926 05:48:24.731765 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4472,"thread_start_us":557,"threads_started":1}
I20260926 05:48:24.732712 23817 heartbeater.cc:347] Connected to a master server at 127.11.247.62:43797
I20260926 05:48:24.732807 23817 heartbeater.cc:464] Registering TS with master...
I20260926 05:48:24.733011 23817 heartbeater.cc:511] Master 127.11.247.62:43797 requested a full tablet report, sending...
I20260926 05:48:24.733420 23467 ts_manager.cc:194] Registered new tserver with Master: e6a6dea3e54f433fbf07b283f9fcac9d (127.11.247.2:43453)
I20260926 05:48:24.733911 23467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:57101
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:48:24.761984 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:33555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:43797
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:48:24.883127 23822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:48:24.883349 23822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:48:24.883374 23822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:48:24.884622 23822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:48:24.884672 23822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:48:24.884701 23822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:48:24.886269 23822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.247.20:33555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:48:24.886526 23822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:48:24.886868 23822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:48:24.893232 23828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:48:24.893255 23827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:48:24.893250 23830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:48:24.893796 23822 server_base.cc:1034] running on GCE node
I20260926 05:48:24.894057 23822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:48:24.894361 23822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:48:24.895522 23822 hybrid_clock.cc:648] HybridClock initialized: now 1790401704895509 us; error 28 us; skew 500 ppm
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:48:24.902412 23822 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:48:24.903041 23822 webserver.cc:460] Webserver started at http://127.11.247.3:39849/ using document root <none> and password file <none>
I20260926 05:48:24.903296 23822 fs_manager.cc:359] Metadata directory not provided
I20260926 05:48:24.903369 23822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:48:24.903529 23822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:48:24 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:48:24.916894 23822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "86e04aa408234d88855e212f2fb3be2c"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "79fef0cc1afa2d34752c2d7fc70e0a45"
server_key_iv: "0e24abd3ea172992de92e9e37bb99f66"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.917495 23822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "86e04aa408234d88855e212f2fb3be2c"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "79fef0cc1afa2d34752c2d7fc70e0a45"
server_key_iv: "0e24abd3ea172992de92e9e37bb99f66"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.929634 23822 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.002s sys 0.000s
I20260926 05:48:24.965449 23837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:24.966614 23822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 05:48:24.966753 23822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "86e04aa408234d88855e212f2fb3be2c"
format_stamp: "Formatted at 2026-09-26 05:48:24 on dist-test-slave-21p4"
server_key: "79fef0cc1afa2d34752c2d7fc70e0a45"
server_key_iv: "0e24abd3ea172992de92e9e37bb99f66"
server_key_version: "kuduclusterkey@0"
I20260926 05:48:24.966881 23822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:48:24.997488 23822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:48:25.000950 23822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:48:25.001190 23822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:48:25.001519 23822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:48:25.001962 23822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:48:25.002058 23822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:25.002137 23822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:48:25.002204 23822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:48:25.009519 23822 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:38685
I20260926 05:48:25.010149 23822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:48:25.014079 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 23822
I20260926 05:48:25.014209 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:48:25.014416 12252 external_mini_cluster.cc:1428] Setting key 53d4dae630d0071e5f060755ed24206f
I20260926 05:48:25.015844 23950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:38685 every 8 connection(s)
Sep 26 05:48:25 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:48:25.031286 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.017186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:42283 (local address 127.11.247.62:43797)
0926 05:48:25.017231 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.017234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.017573 (+ 339us) server_negotiation.cc:407] Connection header received
0926 05:48:25.018302 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.018304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.018317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.018351 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:48:25.019389 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.020202 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.022192 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.022523 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.022607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.026795 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:48:25.026797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:48:25.026797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:48:25.026806 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:48:25.029348 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:25.029981 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:25.029983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:25.029984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:25.030053 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:25.030384 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:25.030386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:25.030386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:25.030558 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:48:25.030640 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.031062 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.031171 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:48:25.032415 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.016884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:43797 (local address 127.11.247.3:42283)
0926 05:48:25.017438 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.017491 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:25.018158 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:25.018555 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:25.018560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:25.018864 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:48:25.019169 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.019176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.020555 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.020556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.022080 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.022083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.022712 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.022713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.022864 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.024183 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:48:25.024194 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:48:25.026659 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:48:25.029479 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:48:25.029489 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:48:25.029492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:48:25.029874 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:48:25.030135 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:48:25.030136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:48:25.030136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:48:25.030301 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:48:25.030691 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:48:25.030698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:48:25.030888 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 05:48:25.031950 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0926 05:48:25.032147 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":100,"threads_started":1}
I20260926 05:48:25.033164 23951 heartbeater.cc:347] Connected to a master server at 127.11.247.62:43797
I20260926 05:48:25.033269 23951 heartbeater.cc:464] Registering TS with master...
I20260926 05:48:25.033489 23951 heartbeater.cc:511] Master 127.11.247.62:43797 requested a full tablet report, sending...
I20260926 05:48:25.033948 23467 ts_manager.cc:194] Registered new tserver with Master: 86e04aa408234d88855e212f2fb3be2c (127.11.247.3:38685)
I20260926 05:48:25.034490 23467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:42283
I20260926 05:48:25.037429 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:48:25.048540 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.039171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.11.247.62:43797)
0926 05:48:25.039233 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.039235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.039480 (+ 245us) server_negotiation.cc:407] Connection header received
0926 05:48:25.039580 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.039580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.039592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.039622 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:48:25.040386 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.041178 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.042317 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.042682 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.042727 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.045460 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:48:25.045461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:48:25.045462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:48:25.045470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:48:25.047235 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:25.047650 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:25.047651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:25.047651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:25.047706 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:48:25.047903 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:48:25.047904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:48:25.047904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:48:25.048061 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:48:25.048138 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.048358 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.048438 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:48:25.050958 23467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:48:25.053182 23467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:48:25.084765 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.071546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:43453 (local address 127.0.0.1:48558)
0926 05:48:25.079210 (+ 7664us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.079219 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:25.079263 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:25.079619 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:25.079623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:25.079625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:48:25.079716 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.079719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.081216 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.081217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.082720 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.082723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.084439 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.084440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.084523 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.084539 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:48:25.084608 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:48:25.084666 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7649,"thread_start_us":70,"threads_started":1}
I20260926 05:48:25.085752 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.071760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.11.247.2:43453)
0926 05:48:25.078712 (+ 6952us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.078719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.079331 (+ 612us) server_negotiation.cc:407] Connection header received
0926 05:48:25.079396 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.079399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.079417 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.079496 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:48:25.079804 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.081102 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.082848 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.083738 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.083819 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.083919 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.085619 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.085667 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6892,"thread_start_us":78,"threads_started":1}
I20260926 05:48:25.087445 23738 tablet_service.cc:1467] Processing CreateTablet for tablet 05a01d7e5cda440c8419ac98630387da (DEFAULT_TABLE table=test-table [id=d224e20c696c465fafe879d5f2fd9cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:48:25.087855 23738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a01d7e5cda440c8419ac98630387da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:48:25.090984 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.071308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:38685 (local address 127.0.0.1:43594)
0926 05:48:25.079081 (+ 7773us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.079091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:25.079130 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:25.086943 (+ 7813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:25.086944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:25.086948 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:48:25.086994 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.086995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.088524 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.088525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.089657 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.089660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.090739 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.090740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.090811 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.090815 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:48:25.090847 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 05:48:25.090890 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7755,"thread_start_us":94,"threads_started":1}
I20260926 05:48:25.091357 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.071460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.11.247.3:38685)
0926 05:48:25.086700 (+ 15240us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.086706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.086722 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:48:25.086765 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.086767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.086784 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.087070 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:48:25.087158 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.088212 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.089793 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.090635 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.091107 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.091206 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.091235 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.091277 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15192,"thread_start_us":97,"threads_started":1}
I20260926 05:48:25.092053 23885 tablet_service.cc:1467] Processing CreateTablet for tablet 05a01d7e5cda440c8419ac98630387da (DEFAULT_TABLE table=test-table [id=d224e20c696c465fafe879d5f2fd9cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:48:25.092423 23885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a01d7e5cda440c8419ac98630387da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:48:25.094380 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.071300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:44721 (local address 127.0.0.1:32964)
0926 05:48:25.078817 (+ 7517us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.078881 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:25.078957 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:25.084279 (+ 5322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:25.084280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:25.084283 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:48:25.084333 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.084334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.086162 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.086164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.092995 (+ 6831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.092998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.094086 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.094087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.094159 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.094165 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:48:25.094195 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:48:25.094239 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7469,"thread_start_us":90,"threads_started":1}
I20260926 05:48:25.094998 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.071572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.11.247.1:44721)
0926 05:48:25.084019 (+ 12447us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.084026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.084038 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:48:25.084077 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.084079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.084100 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.084185 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:48:25.084434 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.085512 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.093117 (+ 7605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.093984 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.094758 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.094841 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.094871 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.094917 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12400,"thread_start_us":56,"threads_started":1}
I20260926 05:48:25.095786 23611 tablet_service.cc:1467] Processing CreateTablet for tablet 05a01d7e5cda440c8419ac98630387da (DEFAULT_TABLE table=test-table [id=d224e20c696c465fafe879d5f2fd9cc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:48:25.096122 23611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a01d7e5cda440c8419ac98630387da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:48:25.096156 23970 tablet_bootstrap.cc:492] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Bootstrap starting.
I20260926 05:48:25.097208 23970 tablet_bootstrap.cc:654] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Neither blocks nor log segments found. Creating new log.
I20260926 05:48:25.097533 23970 log.cc:824] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:48:25.097955 23969 tablet_bootstrap.cc:492] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Bootstrap starting.
I20260926 05:48:25.099040 23969 tablet_bootstrap.cc:654] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Neither blocks nor log segments found. Creating new log.
I20260926 05:48:25.099344 23969 log.cc:824] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:48:25.102878 23972 tablet_bootstrap.cc:492] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: Bootstrap starting.
I20260926 05:48:25.103888 23972 tablet_bootstrap.cc:654] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: Neither blocks nor log segments found. Creating new log.
I20260926 05:48:25.104707 23969 tablet_bootstrap.cc:492] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: No bootstrap required, opened a new log
I20260926 05:48:25.105024 23969 ts_tablet_manager.cc:1397] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 05:48:25.107152 23969 raft_consensus.cc:348] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.108668 23972 log.cc:824] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:48:25.109030 23969 raft_consensus.cc:374] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:48:25.109148 23969 raft_consensus.cc:729] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a6dea3e54f433fbf07b283f9fcac9d, State: Initialized, Role: FOLLOWER
I20260926 05:48:25.109380 23969 consensus_queue.cc:260] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.109795 23969 ts_tablet_manager.cc:1428] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 05:48:25.109855 23970 tablet_bootstrap.cc:492] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: No bootstrap required, opened a new log
I20260926 05:48:25.109964 23817 heartbeater.cc:503] Master 127.11.247.62:43797 was elected leader, sending a full tablet report...
I20260926 05:48:25.110037 23970 ts_tablet_manager.cc:1397] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260926 05:48:25.112285 23970 raft_consensus.cc:348] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.115401 23972 tablet_bootstrap.cc:492] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: No bootstrap required, opened a new log
I20260926 05:48:25.115509 23972 ts_tablet_manager.cc:1397] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260926 05:48:25.115741 23970 raft_consensus.cc:374] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:48:25.115886 23970 raft_consensus.cc:729] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e04aa408234d88855e212f2fb3be2c, State: Initialized, Role: FOLLOWER
I20260926 05:48:25.116132 23970 consensus_queue.cc:260] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.118698 23972 raft_consensus.cc:348] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.118978 23972 raft_consensus.cc:374] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:48:25.119059 23972 raft_consensus.cc:729] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14960e65a17465db4484efac42c97a3, State: Initialized, Role: FOLLOWER
I20260926 05:48:25.119254 23972 consensus_queue.cc:260] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.119657 23972 ts_tablet_manager.cc:1428] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:48:25.119822 23678 heartbeater.cc:503] Master 127.11.247.62:43797 was elected leader, sending a full tablet report...
I20260926 05:48:25.122366 23970 ts_tablet_manager.cc:1428] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20260926 05:48:25.122618 23951 heartbeater.cc:503] Master 127.11.247.62:43797 was elected leader, sending a full tablet report...
Sep 26 05:48:25 dist-test-slave-21p4 krb5kdc[22991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401704, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:48:25.195417 23976 raft_consensus.cc:482] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:48:25.195576 23976 raft_consensus.cc:504] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.195969 23976 leader_election.cc:288] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86e04aa408234d88855e212f2fb3be2c (127.11.247.3:38685), e6a6dea3e54f433fbf07b283f9fcac9d (127.11.247.2:43453)
W20260926 05:48:25.203037 23679 tablet.cc:2369] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:48:25.213953 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.196121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:38685 (local address 127.11.247.1:35265)
0926 05:48:25.206893 (+ 10772us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.206913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:25.207221 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:25.207567 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:25.207568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:25.207570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:48:25.207665 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.207667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.211509 (+ 3842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.211510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.212685 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.212687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.213518 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.213519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.213596 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.213600 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:48:25.213801 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 05:48:25.213848 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10741,"thread_start_us":130,"threads_started":1}
I20260926 05:48:25.215098 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.196361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:35265 (local address 127.11.247.3:38685)
0926 05:48:25.196411 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.196414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.207022 (+ 10608us) server_negotiation.cc:407] Connection header received
0926 05:48:25.207301 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.207302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.207312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.207344 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:48:25.210559 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.211377 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.212792 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.213349 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.213404 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.213415 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.214976 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.215030 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:48:25.215523 23905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a01d7e5cda440c8419ac98630387da" candidate_uuid: "b14960e65a17465db4484efac42c97a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e04aa408234d88855e212f2fb3be2c" is_pre_election: true
I20260926 05:48:25.215709 23905 raft_consensus.cc:2454] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14960e65a17465db4484efac42c97a3 in term 0.
I20260926 05:48:25.215785 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.196379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:55691 (local address 127.11.247.2:43453)
0926 05:48:25.196436 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.196438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.207121 (+ 10683us) server_negotiation.cc:407] Connection header received
0926 05:48:25.207420 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.207421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.207435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.207491 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:48:25.208147 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.208946 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.213700 (+ 4754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.214280 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.214365 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.214374 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.215658 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.215711 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:48:25.216133 23566 leader_election.cc:302] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86e04aa408234d88855e212f2fb3be2c, b14960e65a17465db4484efac42c97a3; no voters:
I20260926 05:48:25.216284 23976 raft_consensus.cc:2781] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:48:25.216327 23976 raft_consensus.cc:482] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:48:25.216349 23976 raft_consensus.cc:3037] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:48:25.217267 23976 raft_consensus.cc:504] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.217407 23976 leader_election.cc:288] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [CANDIDATE]: Term 1 election: Requested vote from peers 86e04aa408234d88855e212f2fb3be2c (127.11.247.3:38685), e6a6dea3e54f433fbf07b283f9fcac9d (127.11.247.2:43453)
W20260926 05:48:25.217564 23818 tablet.cc:2369] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:48:25.218107 23905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a01d7e5cda440c8419ac98630387da" candidate_uuid: "b14960e65a17465db4484efac42c97a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86e04aa408234d88855e212f2fb3be2c"
I20260926 05:48:25.218194 23905 raft_consensus.cc:3037] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:48:25.219195 23905 raft_consensus.cc:2454] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14960e65a17465db4484efac42c97a3 in term 1.
I20260926 05:48:25.219439 23566 leader_election.cc:302] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86e04aa408234d88855e212f2fb3be2c, b14960e65a17465db4484efac42c97a3; no voters:
I20260926 05:48:25.219568 23976 raft_consensus.cc:2781] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:48:25.219753 23976 raft_consensus.cc:686] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [term 1 LEADER]: Becoming Leader. State: Replica: b14960e65a17465db4484efac42c97a3, State: Running, Role: LEADER
I20260926 05:48:25.219852 23976 consensus_queue.cc:237] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } }
I20260926 05:48:25.220820 23467 catalog_manager.cc:5495] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b14960e65a17465db4484efac42c97a3 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "b14960e65a17465db4484efac42c97a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14960e65a17465db4484efac42c97a3" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 } health_report { overall_health: UNKNOWN } } }
I20260926 05:48:25.222262 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.196172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:43453 (local address 127.11.247.1:55691)
0926 05:48:25.207063 (+ 10891us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.207073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:48:25.207171 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:48:25.207765 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:48:25.207766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:48:25.207767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:48:25.208010 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.208013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.209075 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.209076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.210436 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:48:25.210439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.215471 (+ 5032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.215472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:48:25.215554 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.215559 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:48:25.222092 (+ 6533us) client_negotiation.cc:240] Negotiation successful
0926 05:48:25.222153 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10880,"thread_start_us":109,"threads_started":1}
I20260926 05:48:25.222781 23760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a01d7e5cda440c8419ac98630387da" candidate_uuid: "b14960e65a17465db4484efac42c97a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" is_pre_election: true
I20260926 05:48:25.222970 23760 raft_consensus.cc:2454] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14960e65a17465db4484efac42c97a3 in term 0.
I20260926 05:48:25.223100 23770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a01d7e5cda440c8419ac98630387da" candidate_uuid: "b14960e65a17465db4484efac42c97a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d"
I20260926 05:48:25.223165 23770 raft_consensus.cc:3037] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:48:25.224042 23770 raft_consensus.cc:2454] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14960e65a17465db4484efac42c97a3 in term 1.
I20260926 05:48:25.230005 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.226685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.11.247.1:44721)
0926 05:48:25.226737 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:48:25.226740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:48:25.226868 (+ 128us) server_negotiation.cc:407] Connection header received
0926 05:48:25.226941 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:48:25.226941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:48:25.226960 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:48:25.227027 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:48:25.227205 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.227905 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.228974 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:48:25.229241 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:48:25.229296 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:48:25.229484 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:48:25.229607 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:48:25.229673 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:48:25.229875 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 05:48:25.229896 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:48:25.232151 23770 raft_consensus.cc:1261] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 1 FOLLOWER]: Refusing update from remote peer b14960e65a17465db4484efac42c97a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:48:25.232549 23980 consensus_queue.cc:1035] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a6dea3e54f433fbf07b283f9fcac9d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:48:25.234663 23905 raft_consensus.cc:1261] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 1 FOLLOWER]: Refusing update from remote peer b14960e65a17465db4484efac42c97a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:48:25.235189 23980 consensus_queue.cc:1035] T 05a01d7e5cda440c8419ac98630387da P b14960e65a17465db4484efac42c97a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86e04aa408234d88855e212f2fb3be2c" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:48:25.239655 23983 mvcc.cc:205] Tried to move back new op lower bound from 7333485384628006912 to 7333485384581046272. Current Snapshot: MvccSnapshot[applied={T|T < 7333485384628006912}]
I20260926 05:48:25.239818 23984 mvcc.cc:205] Tried to move back new op lower bound from 7333485384628006912 to 7333485384581046272. Current Snapshot: MvccSnapshot[applied={T|T < 7333485384628006912}]
I20260926 05:48:25.242300 23986 mvcc.cc:205] Tried to move back new op lower bound from 7333485384628006912 to 7333485384581046272. Current Snapshot: MvccSnapshot[applied={T|T < 7333485384628006912}]
I20260926 05:48:25.245390 23467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:48:25.245494 23467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:48:25.246292 23467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3fdfa3cffc3451ba768b785c1b1d040: Sending DeleteTablet for 3 replicas of tablet 05a01d7e5cda440c8419ac98630387da
I20260926 05:48:25.246671 23738 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a01d7e5cda440c8419ac98630387da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:48:25 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:48558
I20260926 05:48:25.246706 23885 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a01d7e5cda440c8419ac98630387da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:48:25 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:43594
I20260926 05:48:25.247064 23611 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a01d7e5cda440c8419ac98630387da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:48:25 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:32964
I20260926 05:48:25.247694 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 23548
I20260926 05:48:25.254972 23992 tablet_replica.cc:330] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: stopping tablet replica
I20260926 05:48:25.255138 23992 raft_consensus.cc:2229] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:48:25.255267 23992 raft_consensus.cc:2258] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:48:25.255537 23993 tablet_replica.cc:330] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: stopping tablet replica
I20260926 05:48:25.255643 23993 raft_consensus.cc:2229] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:48:25.255736 23993 raft_consensus.cc:2258] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:48:25.256356 23992 ts_tablet_manager.cc:1905] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:48:25.258386 23993 ts_tablet_manager.cc:1905] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:48:25.258682 23992 ts_tablet_manager.cc:1918] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:48:25.258764 23992 log.cc:1196] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/05a01d7e5cda440c8419ac98630387da
I20260926 05:48:25.259038 23992 ts_tablet_manager.cc:1939] T 05a01d7e5cda440c8419ac98630387da P e6a6dea3e54f433fbf07b283f9fcac9d: Deleting consensus metadata
I20260926 05:48:25.259565 23451 catalog_manager.cc:4841] TS e6a6dea3e54f433fbf07b283f9fcac9d (127.11.247.2:43453): tablet 05a01d7e5cda440c8419ac98630387da (table test-table [id=d224e20c696c465fafe879d5f2fd9cc9]) successfully deleted
I20260926 05:48:25.261140 23993 ts_tablet_manager.cc:1918] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:48:25.261240 23993 log.cc:1196] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/05a01d7e5cda440c8419ac98630387da
I20260926 05:48:25.261880 23993 ts_tablet_manager.cc:1939] T 05a01d7e5cda440c8419ac98630387da P 86e04aa408234d88855e212f2fb3be2c: Deleting consensus metadata
I20260926 05:48:25.262369 23453 catalog_manager.cc:4841] TS 86e04aa408234d88855e212f2fb3be2c (127.11.247.3:38685): tablet 05a01d7e5cda440c8419ac98630387da (table test-table [id=d224e20c696c465fafe879d5f2fd9cc9]) successfully deleted
W20260926 05:48:25.262876 23454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:44721 (error 108)
I20260926 05:48:25.263607 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 23685
I20260926 05:48:25.264213 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:48:25.263750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:44721 (local address 127.0.0.1:32984)
0926 05:48:25.264135 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 05:48:25.264176 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
W20260926 05:48:25.264791 23454 catalog_manager.cc:4584] TS b14960e65a17465db4484efac42c97a3 (127.11.247.1:44721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05a01d7e5cda440c8419ac98630387da: Network error: Client connection negotiation failed: client connection to 127.11.247.1:44721: connect: Connection refused (error 111)
I20260926 05:48:25.270140 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 23822
I20260926 05:48:25.276196 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 23435
I20260926 05:48:25.281862 12252 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 05:48:25.626891 12252 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 05:48:25.627035 12252 mini_ranger.cc:66] Stopping Ranger...
I20260926 05:48:25.972100 12252 mini_ranger.cc:68] Stopped Ranger
2026-09-26 05:48:25.972 UTC [23021] LOG: received smart shutdown request
2026-09-26 05:48:25.976 UTC [23021] LOG: background worker "logical replication launcher" (PID 23030) exited with exit code 1
2026-09-26 05:48:25.976 UTC [23025] LOG: shutting down
2026-09-26 05:48:26.197 UTC [23021] LOG: database system is shut down
2026-09-26T05:48:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50793 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:48:26 dist-test-slave-21p4 krb5kdc[24004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:48:26 dist-test-slave-21p4 krb5kdc[24004](info): set up 2 sockets
Sep 26 05:48:26 dist-test-slave-21p4 krb5kdc[24004](info): commencing operation
krb5kdc: starting...
W20260926 05:48:27.358533 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:48:27 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:48:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:48:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/rangeradmin_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/rangeradmin_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/rangerlookup_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/rangerlookup_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
I20260926 05:48:27.526502 12252 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksryJ5D/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/postgres -l logfile start
2026-09-26 05:48:30.282 UTC [24034] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 05:48:30.282 UTC [24034] LOG: listening on IPv4 address "127.11.247.20", port 51277
2026-09-26 05:48:30.286 UTC [24034] LOG: listening on Unix socket "/tmp/.s.PGSQL.51277"
2026-09-26 05:48:30.322 UTC [24037] LOG: database system was shut down at 2026-09-26 05:48:28 UTC
2026-09-26 05:48:30.327 UTC [24034] LOG: database system is ready to accept connections
I20260926 05:48:31.270748 12252 mini_postgres.cc:96] Postgres bound to 51277
2026-09-26 05:48:31.276 UTC [24045] FATAL: database "slave" does not exist
127.11.247.20:51277 - accepting connections
I20260926 05:48:31.277513 12252 mini_ranger.cc:161] Starting Ranger...
I20260926 05:48:31.293082 12252 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 05:48:31.881332 12252 mini_ranger.cc:87] Created ranger Postgres database
I20260926 05:48:31.881419 12252 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-admin
2026-09-26 05:48:32,167 [I] DB FLAVOR :POSTGRES
2026-09-26 05:48:32,167 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:48:32,167 [I] Checking connection..
2026-09-26 05:48:32,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 05:48:32,417 [I] Checking connection passed.
2026-09-26 05:48:32,417 [I] --------- Verifying version history table ---------
2026-09-26 05:48:32,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:48:32,666 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:48:32,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:48:32,919 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:48:32,919 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 05:48:32,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 05:48:33.162 UTC [24109] WARNING: there is no transaction in progress
2026-09-26 05:48:33,169 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 05:48:33,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:48:33,422 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 05:48:33,422 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:48:33,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 05:48:33,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 05:48:33,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-21p4.c.gcp-upstream.internal','N') ;"
2026-09-26 05:48:34,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:48:34,433 [I] Table x_portal_user does not exist in database ranger
2026-09-26 05:48:34,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:48:34,669 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 05:48:34,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:48:34,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:48:35,194 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 05:48:35,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 05:48:36.259 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.279 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.298 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.318 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.339 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.360 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.420 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.428 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.444 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.456 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.466 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.483 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.494 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.502 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.510 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:36.518 UTC [24217] WARNING: there is no transaction in progress
2026-09-26 05:48:37,290 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 05:48:37,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 05:48:37,557 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 05:48:37,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:48:37,820 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 05:48:37,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:48:38,072 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 05:48:38,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:48:38,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:48:38,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-21p4.c.gcp-upstream.internal';"
2026-09-26 05:48:38,783 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 05:48:38,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 05:48:39,018 [I] DB_PATCHES have already been applied
I20260926 05:48:39.024173 12252 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksryJ5D/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 05:48:39.024258 12252 mini_ranger.cc:193] Using host: 127.11.247.20
I20260926 05:48:39.027627 12252 mini_ranger.cc:227] Ranger admin URL: http://127.11.247.20:35449
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 5:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 05:48:40.144586 12252 mini_ranger.cc:160] Time spent starting Ranger: real 8.867s user 0.000s sys 0.009s
Sep 26, 2026 5:49:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 5:49:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 5:49:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 5:49:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 5:49:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 05:49:03.362742 12252 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.247.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.247.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/rangerkms_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.247.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/rangerkms_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.247.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.247.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.247.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/HTTP_127.11.247.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 05:49:03 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401743, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 05:49:03.452944 12252 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 05:49:03.463431 12252 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 05:49:04.407562 12252 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 05:49:04.407646 12252 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms
I20260926 05:49:04.407660 12252 mini_ranger_kms.cc:227] Using postgres at 127.11.247.20:51277
2026-09-26 05:49:04,573 [I] DB FLAVOR :POSTGRES
2026-09-26 05:49:04,574 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:49:04,574 [I] Checking connection
2026-09-26 05:49:04,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 05:49:04,848 [I] connection success
2026-09-26 05:49:04,849 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 05:49:04,849 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 05:49:04,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 05:49:05,105 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 05:49:05,105 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:49:05,105 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 05:49:05,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.247.20:51277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 05:49:05,419 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 05:49:05.998497 12252 mini_ranger_kms.cc:310] Created kms service
I20260926 05:49:06.237650 12252 mini_ranger_kms.cc:326] Created kudu user
I20260926 05:49:06.288252 12252 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 05:49:06 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401743, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:06.677804 12252 mini_ranger_kms.cc:384] Added ranger policy
I20260926 05:49:06.678031 12252 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksryJ5D/build/release/bin/postgresql.jar:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksryJ5D/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 05:49:06.678059 12252 mini_ranger_kms.cc:240] Using host: 127.11.247.20
I20260926 05:49:06.682010 12252 mini_ranger_kms.cc:276] Ranger KMS PID: 24398
I20260926 05:49:06.682106 12252 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.247.20:39863
Sep 26, 2026 5:49:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:49:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:49:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 5:49:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 5:49:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 5:49:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 05:49:08.946478 12252 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.494s user 0.000s sys 0.148s
I20260926 05:49:08.946570 12252 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 05:49:08.946638 12252 mini_ranger_kms.cc:401] 127.11.247.20:39863/kms/v1/keys
Sep 26 05:49:09 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.247.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 05:49:09 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.247.20@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
Sep 26 05:49:09 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.247.20@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
Sep 26 05:49:12 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:49:12.210206 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33751
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33751
--ranger_config_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:12.306954 24448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:12.307188 24448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:12.307214 24448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:12.308594 24448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:12.308653 24448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:12.308668 24448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:12.308684 24448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:12.308697 24448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:12.310498 24448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.247.20:39863/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33751
--ranger_config_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:12.310840 24448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:12.311108 24448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:12.316864 24454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:12.317070 24456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:12.316929 24448 server_base.cc:1034] running on GCE node
W20260926 05:49:12.316867 24453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:12.317406 24448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:12.317677 24448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:12.318854 24448 hybrid_clock.cc:648] HybridClock initialized: now 1790401752318839 us; error 30 us; skew 500 ppm
Sep 26 05:49:12 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401752, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:12.321575 24448 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:12.322075 24448 webserver.cc:460] Webserver started at http://127.11.247.62:38943/ using document root <none> and password file <none>
I20260926 05:49:12.322315 24448 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:12.322376 24448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:12.322525 24448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:49:12 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401752, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:12.381834 24448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "dca481ce37c04761b4aeff40f0162237"
format_stamp: "Formatted at 2026-09-26 05:49:12 on dist-test-slave-21p4"
server_key: "cc8f7409bd21827b953ea62d8ad95471"
server_key_iv: "8b8ae2338d5c12938d704262ba8ae800"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:12.382287 24448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "dca481ce37c04761b4aeff40f0162237"
format_stamp: "Formatted at 2026-09-26 05:49:12 on dist-test-slave-21p4"
server_key: "cc8f7409bd21827b953ea62d8ad95471"
server_key_iv: "8b8ae2338d5c12938d704262ba8ae800"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:12.383886 24448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 05:49:12.400019 24463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:12.400282 24448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:12.400444 24448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "dca481ce37c04761b4aeff40f0162237"
format_stamp: "Formatted at 2026-09-26 05:49:12 on dist-test-slave-21p4"
server_key: "cc8f7409bd21827b953ea62d8ad95471"
server_key_iv: "8b8ae2338d5c12938d704262ba8ae800"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:12.400579 24448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:12.425268 24448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:12.427593 24448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:12.427747 24448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:12.432924 24448 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33751
I20260926 05:49:12.432947 24515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33751 every 8 connection(s)
I20260926 05:49:12.433388 24448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:12.434131 24516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:12.436805 24516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Bootstrap starting.
I20260926 05:49:12.437552 24516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:12.437608 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 24448
I20260926 05:49:12.437706 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:49:12.437924 12252 external_mini_cluster.cc:1428] Setting key e6a55e23970ba851bf148c07a0f37e5b
I20260926 05:49:12.437920 24516 log.cc:824] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:12.438740 24516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: No bootstrap required, opened a new log
I20260926 05:49:12.441493 24516 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:12.441682 24516 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:12.441712 24516 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca481ce37c04761b4aeff40f0162237, State: Initialized, Role: FOLLOWER
I20260926 05:49:12.441879 24516 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:12.442016 24516 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:12.442095 24516 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:12.442165 24516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:12.442900 24516 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:12.443029 24516 leader_election.cc:302] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca481ce37c04761b4aeff40f0162237; no voters:
I20260926 05:49:12.443220 24516 leader_election.cc:288] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:49:12.443436 24522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:12.443588 24516 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:12.443663 24522 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 LEADER]: Becoming Leader. State: Replica: dca481ce37c04761b4aeff40f0162237, State: Running, Role: LEADER
I20260926 05:49:12.443805 24522 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:12.444231 24522 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca481ce37c04761b4aeff40f0162237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } } }
I20260926 05:49:12.444423 24522 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:12.444603 24522 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca481ce37c04761b4aeff40f0162237. Latest consensus state: current_term: 1 leader_uuid: "dca481ce37c04761b4aeff40f0162237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } } }
I20260926 05:49:12.444733 24522 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:12.444914 24530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:12.445188 24516 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:49:12.445498 24530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:49:12 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:12.452571 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:12.438492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.11.247.62:33751)
0926 05:49:12.438705 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:12.438714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:12.438747 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:49:12.439314 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:12.439323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:12.439392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:12.439557 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:12.440341 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:12.441859 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:12.443036 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:12.443706 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:12.443826 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:12.447414 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:12.447464 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:12.447464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:12.447581 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:12.450889 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:12.451305 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:12.451309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:12.451309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:12.451405 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:12.451619 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:12.451619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:12.451620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:12.451913 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:12.452088 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:12.452200 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 05:49:12.452364 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260926 05:49:12.453583 24530 catalog_manager.cc:1347] Generated new cluster ID: e27e21d285ef402fb90612a4c7ed78a1
I20260926 05:49:12.453658 24530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:49:12.471578 24530 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:49:12.472503 24530 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:12.478520 24530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Generated new TSK 0
I20260926 05:49:12.478904 24530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:49:13.249379 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33751
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:13.362277 24561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:13.362486 24561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:13.362511 24561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:13.363726 24561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:13.363801 24561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:13.363847 24561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:49:13.365834 24561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.247.20:39863/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:13.366118 24561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:13.366443 24561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:13.372202 24566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:13.372280 24561 server_base.cc:1034] running on GCE node
W20260926 05:49:13.372395 24567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:13.372368 24569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:13.372709 24561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:13.373052 24561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:13.374226 24561 hybrid_clock.cc:648] HybridClock initialized: now 1790401753374206 us; error 39 us; skew 500 ppm
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:13.377103 24561 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:49:13.377597 24561 webserver.cc:460] Webserver started at http://127.11.247.1:44593/ using document root <none> and password file <none>
I20260926 05:49:13.377835 24561 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:13.377905 24561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:13.378060 24561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:13.393581 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "215f186c37e24257b0ddcc9274340c9d"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "6aee97600e8ff9e4029e5259ef4b1559"
server_key_iv: "925f1e1317207f4ee2199268816b8c13"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.394170 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "215f186c37e24257b0ddcc9274340c9d"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "6aee97600e8ff9e4029e5259ef4b1559"
server_key_iv: "925f1e1317207f4ee2199268816b8c13"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.395967 24561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:13.409160 24578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.409395 24561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:13.409503 24561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "215f186c37e24257b0ddcc9274340c9d"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "6aee97600e8ff9e4029e5259ef4b1559"
server_key_iv: "925f1e1317207f4ee2199268816b8c13"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.409618 24561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:13.425993 24561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:13.428203 24561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:13.428381 24561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:13.428661 24561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:13.429065 24561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:13.429121 24561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.429157 24561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:13.429203 24561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.437041 24561 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:46569
I20260926 05:49:13.437127 24691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:46569 every 8 connection(s)
I20260926 05:49:13.437455 24561 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:49:13.441143 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 24561
I20260926 05:49:13.441313 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:49:13.441532 12252 external_mini_cluster.cc:1428] Setting key 40c4bd4a24a5d3ce28b47873c5613f73
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:13.449941 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:13.438264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33751 (local address 127.11.247.1:39785)
0926 05:49:13.438453 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:13.438504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:13.439261 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:13.439589 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:13.439593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:13.439901 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:13.440200 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:13.440207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.441270 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.441272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:13.442619 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:13.442622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.443224 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.443226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:13.443378 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:13.444099 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:13.444111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:13.445733 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:13.448010 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:13.448017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:13.448021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:13.448364 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:13.448617 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:13.448617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:13.448618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:13.448784 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:13.449111 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:13.449118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:13.449311 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 05:49:13.449416 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 05:49:13.449611 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260926 05:49:13.450369 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:13.438262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:39785 (local address 127.11.247.62:33751)
0926 05:49:13.438537 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:13.438539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:13.438651 (+ 112us) server_negotiation.cc:407] Connection header received
0926 05:49:13.439400 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:13.439401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:13.439415 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:13.439496 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:13.440325 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.441139 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.442727 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.443074 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.443139 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:13.445850 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:13.445852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:13.445853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:13.445861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:13.447903 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:13.448463 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:13.448465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:13.448465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:13.448538 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:13.448867 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:13.448868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:13.448868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:13.449030 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:13.450115 (+ 1085us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:13.450170 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:49:13.450260 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":85,"threads_started":1}
I20260926 05:49:13.450898 24692 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33751
I20260926 05:49:13.451017 24692 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:13.451236 24692 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:13.451855 24480 ts_manager.cc:194] Registered new tserver with Master: 215f186c37e24257b0ddcc9274340c9d (127.11.247.1:46569)
I20260926 05:49:13.452529 24480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:39785
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:49:13.474838 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33751
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:13.591581 24698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:13.591958 24698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:13.592048 24698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 05:49:13.594004 24698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:13.594112 24698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:13.594213 24698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:49:13.596696 24698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.247.20:39863/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:13.597216 24698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:13.597606 24698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:13.604547 24708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:13.604856 24698 server_base.cc:1034] running on GCE node
W20260926 05:49:13.605060 24705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:13.605204 24706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:13.605527 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:13.605849 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:13.607019 24698 hybrid_clock.cc:648] HybridClock initialized: now 1790401753606996 us; error 36 us; skew 500 ppm
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:13.610412 24698 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:49:13.610975 24698 webserver.cc:460] Webserver started at http://127.11.247.2:42661/ using document root <none> and password file <none>
I20260926 05:49:13.611238 24698 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:13.611289 24698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:13.611411 24698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:13.632390 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "df43c43530064d11a6afbae7e36199a4"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "9670018613884dfb94a033ec08383121"
server_key_iv: "8aa1b638ab286c48ff2b62b0390bb81b"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.632861 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "df43c43530064d11a6afbae7e36199a4"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "9670018613884dfb94a033ec08383121"
server_key_iv: "8aa1b638ab286c48ff2b62b0390bb81b"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.639089 24698 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 05:49:13.654832 24715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.655098 24698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:49:13.655229 24698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "df43c43530064d11a6afbae7e36199a4"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "9670018613884dfb94a033ec08383121"
server_key_iv: "8aa1b638ab286c48ff2b62b0390bb81b"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.655340 24698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:13.688210 24698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:13.690780 24698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:13.690958 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:13.691214 24698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:13.691630 24698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:13.691682 24698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.691715 24698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:13.691776 24698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.698968 24698 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:46693
I20260926 05:49:13.699424 24698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:49:13.699714 24829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:46693 every 8 connection(s)
I20260926 05:49:13.703380 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 24698
I20260926 05:49:13.703485 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:49:13.703701 12252 external_mini_cluster.cc:1428] Setting key bc5a2bac39a267d1be8a19c622121b0b
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:13.711582 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:13.700219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:38995 (local address 127.11.247.62:33751)
0926 05:49:13.700297 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:13.700300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:13.700559 (+ 259us) server_negotiation.cc:407] Connection header received
0926 05:49:13.701255 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:13.701256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:13.701269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:13.701314 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:13.702100 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.702764 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.704228 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.704561 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.704634 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:13.708387 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:13.708389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:13.708389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:13.708398 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:13.710029 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:13.710479 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:13.710480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:13.710480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:13.710544 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:13.710851 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:13.710852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:13.710852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:13.710991 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:13.711069 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:13.711386 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 05:49:13.711484 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:49:13.711759 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:13.700189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33751 (local address 127.11.247.2:38995)
0926 05:49:13.700407 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:13.700459 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:13.701146 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:13.701389 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:13.701394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:13.701650 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:13.701933 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:13.701938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.702862 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.702863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:13.704122 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:13.704125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.704668 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.704669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:13.704827 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:13.705427 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:13.705438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:13.708238 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:13.710141 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:13.710149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:13.710151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:13.710397 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:13.710649 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:13.710650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:13.710650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:13.710781 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:13.711060 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:13.711067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:13.711228 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 05:49:13.711342 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 05:49:13.711504 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
I20260926 05:49:13.712318 24830 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33751
I20260926 05:49:13.712443 24830 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:13.712682 24830 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:13.713081 24480 ts_manager.cc:194] Registered new tserver with Master: df43c43530064d11a6afbae7e36199a4 (127.11.247.2:46693)
I20260926 05:49:13.713476 24480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:38995
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:49:13.734726 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33751
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:13.860749 24835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:13.861122 24835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:13.861217 24835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:13.863117 24835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:13.863261 24835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:13.863349 24835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:49:13.865644 24835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.247.20:39863/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:13.866133 24835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:13.866492 24835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:13.873471 24841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:13.873878 24840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:13.874141 24843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:13.875170 24835 server_base.cc:1034] running on GCE node
I20260926 05:49:13.875581 24835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:13.875872 24835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:13.879226 24835 hybrid_clock.cc:648] HybridClock initialized: now 1790401753879211 us; error 54 us; skew 500 ppm
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:13.882905 24835 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:49:13.883484 24835 webserver.cc:460] Webserver started at http://127.11.247.3:37269/ using document root <none> and password file <none>
I20260926 05:49:13.883755 24835 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:13.883816 24835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:13.883992 24835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:13.902679 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "9f38aad9c562448f84801972553b82f5"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "de20ad184cd6d188224b5d86eadd02aa"
server_key_iv: "b49d25d6444bcbf62b2db60f6804d11e"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.903368 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "9f38aad9c562448f84801972553b82f5"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "de20ad184cd6d188224b5d86eadd02aa"
server_key_iv: "b49d25d6444bcbf62b2db60f6804d11e"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.907033 24835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 05:49:13.928061 24850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.928371 24835 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 05:49:13.928483 24835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "9f38aad9c562448f84801972553b82f5"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "de20ad184cd6d188224b5d86eadd02aa"
server_key_iv: "b49d25d6444bcbf62b2db60f6804d11e"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:13.928591 24835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:13.946741 24835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:13.949434 24835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:13.949662 24835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:13.950074 24835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:13.950619 24835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:13.950716 24835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.950795 24835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:13.950855 24835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:13.958456 24835 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:45611
I20260926 05:49:13.959002 24835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:49:13.959506 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:45611 every 8 connection(s)
Sep 26 05:49:13 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:13.968115 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 24835
I20260926 05:49:13.968225 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:49:13.968422 12252 external_mini_cluster.cc:1428] Setting key f40a873266fcfba2086177acc0f72880
I20260926 05:49:13.972246 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:13.960586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:46999 (local address 127.11.247.62:33751)
0926 05:49:13.960636 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:13.960638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:13.961041 (+ 403us) server_negotiation.cc:407] Connection header received
0926 05:49:13.961738 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:13.961739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:13.961755 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:13.961848 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:13.962690 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.963503 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.965056 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.965382 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.965432 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:13.968373 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:13.968375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:13.968375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:13.968384 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:13.970472 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:13.971034 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:13.971036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:13.971036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:13.971101 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:13.971438 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:13.971439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:13.971440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:13.971601 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:13.971676 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:13.972058 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 05:49:13.972149 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:49:13.972818 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:13.960290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33751 (local address 127.11.247.3:46999)
0926 05:49:13.960870 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:13.960920 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:13.961599 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:13.961948 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:13.961954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:13.962225 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:13.962541 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:13.962548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.963625 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.963626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:13.964940 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:13.964943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:13.965549 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:13.965550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:13.965699 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:13.966361 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:13.966373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:13.968173 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:13.970569 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:13.970603 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:13.970606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:13.970934 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:13.971181 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:13.971181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:13.971181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:13.971322 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:13.971690 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:13.971697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:13.971881 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 05:49:13.972398 (+ 517us) client_negotiation.cc:240] Negotiation successful
0926 05:49:13.972578 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":87,"threads_started":1}
I20260926 05:49:13.973376 24964 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33751
I20260926 05:49:13.973489 24964 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:13.973690 24964 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:13.974139 24480 ts_manager.cc:194] Registered new tserver with Master: 9f38aad9c562448f84801972553b82f5 (127.11.247.3:45611)
I20260926 05:49:13.974754 24480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:46999
I20260926 05:49:13.980311 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:49:13.981213 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 24561
I20260926 05:49:13.988219 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 24698
I20260926 05:49:13.994719 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 24835
I20260926 05:49:14.001955 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 24448
I20260926 05:49:14.008339 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33751
--webserver_interface=127.11.247.62
--webserver_port=38943
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33751
--ranger_config_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:14.091178 24967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:14.091406 24967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:14.091428 24967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:14.092609 24967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:14.092661 24967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:14.092675 24967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:14.092689 24967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:14.092703 24967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:14.094307 24967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33751
--ranger_config_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:14.094552 24967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:14.094851 24967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:14.099992 24973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:14.100080 24967 server_base.cc:1034] running on GCE node
W20260926 05:49:14.099990 24975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.100003 24972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:14.100514 24967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:14.100793 24967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:14.101940 24967 hybrid_clock.cc:648] HybridClock initialized: now 1790401754101929 us; error 31 us; skew 500 ppm
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:14.104607 24967 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:14.105077 24967 webserver.cc:460] Webserver started at http://127.11.247.62:38943/ using document root <none> and password file <none>
I20260926 05:49:14.105312 24967 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:14.105367 24967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:14.152345 24967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:14.153206 24983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.153523 24967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:14.153648 24967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "dca481ce37c04761b4aeff40f0162237"
format_stamp: "Formatted at 2026-09-26 05:49:12 on dist-test-slave-21p4"
server_key: "cc8f7409bd21827b953ea62d8ad95471"
server_key_iv: "8b8ae2338d5c12938d704262ba8ae800"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:14.154055 24967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:14.170234 24967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:14.172199 24967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:14.172365 24967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:14.176942 24967 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33751
I20260926 05:49:14.176985 25035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33751 every 8 connection(s)
I20260926 05:49:14.177359 24967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:14.178714 25036 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:49:14.179323 25036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Bootstrap starting.
I20260926 05:49:14.184325 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 24967
I20260926 05:49:14.184643 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:46569
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:33751
--webserver_port=44593
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:14.187479 25036 log.cc:824] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:14.189795 25036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:49:14.190115 25036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Bootstrap complete.
I20260926 05:49:14.194399 25036 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:14.194851 25036 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca481ce37c04761b4aeff40f0162237, State: Initialized, Role: FOLLOWER
I20260926 05:49:14.195039 25036 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:14.195128 25036 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:14.195175 25036 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:14.195233 25036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:49:14.196713 25036 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:14.196831 25036 leader_election.cc:302] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca481ce37c04761b4aeff40f0162237; no voters:
I20260926 05:49:14.196993 25036 leader_election.cc:288] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:49:14.197078 25040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:49:14.197238 25036 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:14.197274 25040 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [term 2 LEADER]: Becoming Leader. State: Replica: dca481ce37c04761b4aeff40f0162237, State: Running, Role: LEADER
I20260926 05:49:14.197513 25040 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } }
I20260926 05:49:14.197860 25041 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dca481ce37c04761b4aeff40f0162237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } } }
I20260926 05:49:14.197939 25041 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:14.198222 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca481ce37c04761b4aeff40f0162237. Latest consensus state: current_term: 2 leader_uuid: "dca481ce37c04761b4aeff40f0162237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca481ce37c04761b4aeff40f0162237" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33751 } } }
I20260926 05:49:14.198282 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:14.198599 25048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:14.198899 25036 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:49:14.199112 25048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:14.199278 25048 catalog_manager.cc:1259] Loaded cluster ID: e27e21d285ef402fb90612a4c7ed78a1
I20260926 05:49:14.199353 25048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:49:14.203974 25048 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:14.204375 25048 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Loaded TSK: 0
I20260926 05:49:14.204720 25048 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:49:14.295809 25038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:14.296139 25038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:14.296204 25038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:14.298063 25038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:14.298123 25038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:14.298166 25038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:49:14.300494 25038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:46569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=44593
--tserver_master_addrs=127.11.247.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:14.300747 25038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:14.301034 25038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:14.308225 25077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.308645 25075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.309616 25074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:14.309940 25038 server_base.cc:1034] running on GCE node
I20260926 05:49:14.310170 25038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:14.310473 25038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:14.311661 25038 hybrid_clock.cc:648] HybridClock initialized: now 1790401754311624 us; error 49 us; skew 500 ppm
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:14.314951 25038 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:49:14.315549 25038 webserver.cc:460] Webserver started at http://127.11.247.1:44593/ using document root <none> and password file <none>
I20260926 05:49:14.315817 25038 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:14.315904 25038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:14.331511 25038 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:14.332715 25085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.332914 25038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:14.333005 25038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "215f186c37e24257b0ddcc9274340c9d"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "6aee97600e8ff9e4029e5259ef4b1559"
server_key_iv: "925f1e1317207f4ee2199268816b8c13"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:14.333400 25038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:14.353863 25038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:14.356439 25038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:14.356666 25038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:14.356966 25038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:14.357350 25038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:14.357447 25038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.357554 25038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:14.357635 25038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.364244 25038 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:46569
I20260926 05:49:14.364557 25198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:46569 every 8 connection(s)
I20260926 05:49:14.364898 25038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:49:14.371235 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25038
I20260926 05:49:14.371563 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:46693
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:33751
--webserver_port=42661
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:14.380244 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.366017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:59705 (local address 127.11.247.62:33751)
0926 05:49:14.369351 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:14.369359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:14.369404 (+ 45us) server_negotiation.cc:407] Connection header received
0926 05:49:14.369530 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:14.369535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:14.369580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:14.369745 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:14.370614 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.371602 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.373189 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.373825 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.373944 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.376674 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:14.376678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:14.376678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:14.376689 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:14.378406 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.378895 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.378901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.378901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.378991 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.379259 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.379260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.379261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.379502 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:14.379643 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:14.379922 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 05:49:14.380034 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3286,"thread_start_us":100,"threads_started":1}
I20260926 05:49:14.380255 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.365712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33751 (local address 127.11.247.1:59705)
0926 05:49:14.366327 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:14.366378 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:14.367063 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:14.369798 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:14.369803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:14.370104 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:14.370420 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:14.370428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.371762 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.371763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:14.373088 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:14.373092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.373961 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.373962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:14.374104 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.375010 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:14.375021 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:14.376530 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:14.378509 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:14.378516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:14.378519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:14.378802 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:14.379070 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:14.379071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:14.379071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:14.379182 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:14.379587 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:14.379593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:14.379746 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 05:49:14.379851 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 05:49:14.380020 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":538,"threads_started":1}
I20260926 05:49:14.381102 25199 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33751
I20260926 05:49:14.381201 25199 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.381394 25199 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
W20260926 05:49:14.382251 25199 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:33751 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:49:14.382336 25199 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.382414 25199 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:14.382788 25199 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.382879 25199 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
W20260926 05:49:14.383158 25199 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:49:14.499708 25203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:14.500072 25203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:14.500180 25203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:14.502100 25203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:14.502203 25203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:14.502287 25203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:49:14.504660 25203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:46693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=42661
--tserver_master_addrs=127.11.247.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:14.504935 25203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:14.505264 25203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:14.512892 25209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.512876 25208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.512903 25211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:14.514606 25203 server_base.cc:1034] running on GCE node
I20260926 05:49:14.514900 25203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:14.515233 25203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:14.516471 25203 hybrid_clock.cc:648] HybridClock initialized: now 1790401754516444 us; error 47 us; skew 500 ppm
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:14.519699 25203 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:49:14.520296 25203 webserver.cc:460] Webserver started at http://127.11.247.2:42661/ using document root <none> and password file <none>
I20260926 05:49:14.520598 25203 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:14.520691 25203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:14.540230 25203 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 05:49:14.541193 25219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.541426 25203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:14.541549 25203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "df43c43530064d11a6afbae7e36199a4"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "9670018613884dfb94a033ec08383121"
server_key_iv: "8aa1b638ab286c48ff2b62b0390bb81b"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:14.542064 25203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:14.577765 25203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:14.580354 25203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:14.580575 25203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:14.580878 25203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:14.581310 25203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:14.581365 25203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.581401 25203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:14.581420 25203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.589264 25203 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:46693
I20260926 05:49:14.589510 25332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:46693 every 8 connection(s)
I20260926 05:49:14.589859 25203 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:49:14.590497 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25203
I20260926 05:49:14.590971 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.247.20:39863/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:45611
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:33751
--webserver_port=37269
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:14.602468 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.590851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:60079 (local address 127.11.247.62:33751)
0926 05:49:14.590908 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:14.590910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:14.591188 (+ 278us) server_negotiation.cc:407] Connection header received
0926 05:49:14.592103 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:14.592105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:14.592121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:14.592156 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:14.593228 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.594152 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.595488 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.595709 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.595816 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.598803 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:14.598806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:14.598806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:14.598817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:14.600712 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.601330 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.601331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.601332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.601395 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.601723 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.601725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.601725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.601864 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:14.601947 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:14.602261 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 05:49:14.602362 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:49:14.602701 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.590845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33751 (local address 127.11.247.2:60079)
0926 05:49:14.591033 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:14.591091 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:14.591959 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:14.592370 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:14.592376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:14.592850 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:14.593112 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:14.593118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.594297 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.594298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:14.595368 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:14.595371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.595849 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.595850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:14.595992 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.596636 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:14.596646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:14.598659 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:14.600840 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:14.600850 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:14.600852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:14.601192 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:14.601473 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:14.601474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:14.601474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:14.601638 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:14.601950 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:14.601958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:14.602119 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 05:49:14.602230 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:49:14.602395 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260926 05:49:14.603295 25333 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33751
I20260926 05:49:14.603420 25333 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.603621 25333 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
W20260926 05:49:14.604287 25333 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:33751 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:49:14.604377 25333 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.604455 25333 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:14.604839 25333 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.604918 25333 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
W20260926 05:49:14.605191 25333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:49:14.719640 25336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:14.720016 25336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:14.720105 25336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:14.722057 25336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:14.722180 25336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:14.722290 25336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:49:14.724860 25336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.247.20:39863/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:45611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=37269
--tserver_master_addrs=127.11.247.62:33751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:14.725333 25336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:14.725696 25336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:14.732826 25344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.732878 25342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:14.732872 25341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:14.734174 25336 server_base.cc:1034] running on GCE node
I20260926 05:49:14.734434 25336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:14.734772 25336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:14.738054 25336 hybrid_clock.cc:648] HybridClock initialized: now 1790401754738017 us; error 48 us; skew 500 ppm
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:14.741520 25336 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:49:14.742159 25336 webserver.cc:460] Webserver started at http://127.11.247.3:37269/ using document root <none> and password file <none>
I20260926 05:49:14.742439 25336 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:14.742529 25336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:14.761160 25336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260926 05:49:14.762409 25353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.762749 25336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:49:14.762903 25336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "9f38aad9c562448f84801972553b82f5"
format_stamp: "Formatted at 2026-09-26 05:49:13 on dist-test-slave-21p4"
server_key: "de20ad184cd6d188224b5d86eadd02aa"
server_key_iv: "b49d25d6444bcbf62b2db60f6804d11e"
server_key_version: "kuduclusterkey@0"
I20260926 05:49:14.763451 25336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:14.790390 25336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:14.793957 25336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:14.794174 25336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:14.794464 25336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:14.794929 25336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:14.795017 25336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.795081 25336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:14.795131 25336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:14.802731 25336 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:45611
I20260926 05:49:14.802836 25466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:45611 every 8 connection(s)
I20260926 05:49:14.803191 25336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:49:14 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:14.811802 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25336
I20260926 05:49:14.814955 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.804020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33751 (local address 127.11.247.3:51973)
0926 05:49:14.804181 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:14.804218 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:14.804721 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:14.805945 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:14.805950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:14.806146 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:14.806466 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:14.806472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.807240 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.807242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:14.808108 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:14.808110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.808546 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.808548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:14.808683 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.809316 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:14.809326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:14.810884 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:14.813215 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:14.813224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:14.813227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:14.813572 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:14.813793 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:14.813794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:14.813794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:14.813928 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:14.814195 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:14.814203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:14.814387 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 05:49:14.814505 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 05:49:14.814695 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260926 05:49:14.815613 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.804024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:51973 (local address 127.11.247.62:33751)
0926 05:49:14.804070 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:14.804072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:14.804301 (+ 229us) server_negotiation.cc:407] Connection header received
0926 05:49:14.804868 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:14.804870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:14.804886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:14.804936 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:14.806626 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.807138 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.808209 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.808435 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.808519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.810994 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:14.810996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:14.810996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:14.811004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:14.813116 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.813656 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.813657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.813658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.813718 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.814003 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.814003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.814004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.814123 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:14.814195 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:14.815485 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0926 05:49:14.815552 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:49:14.816011 25467 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33751
I20260926 05:49:14.816133 25467 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.816351 25467 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
W20260926 05:49:14.817173 25467 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:33751 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:49:14.817260 25467 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.817385 25467 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:14.817804 25467 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:14.817901 25467 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
W20260926 05:49:14.818225 25467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:49:14.827224 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:14.812260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.11.247.62:33751)
0926 05:49:14.812585 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:14.812588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:14.812595 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:49:14.812617 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:14.812617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:14.812630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:14.812694 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:14.813524 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.814296 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.818654 (+ 4358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:14.819009 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:14.819086 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:14.820894 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:14.820896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:14.820896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:14.820905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:14.822838 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.824397 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.824399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.824399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.824472 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:14.826322 (+ 1850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:14.826323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:14.826324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:14.826493 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:14.826718 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:14.826986 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 05:49:14.827096 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":74,"threads_started":1}
I20260926 05:49:15.383343 25199 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:15.383502 25199 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:15.383958 25000 ts_manager.cc:194] Registered new tserver with Master: 215f186c37e24257b0ddcc9274340c9d (127.11.247.1:46569)
I20260926 05:49:15.384984 25000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:59705
Sep 26 05:49:15 dist-test-slave-21p4 krb5kdc[24004](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:15.605445 25333 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:15.605630 25333 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:15.606097 25000 ts_manager.cc:194] Registered new tserver with Master: df43c43530064d11a6afbae7e36199a4 (127.11.247.2:46693)
I20260926 05:49:15.606757 25000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:60079
Sep 26 05:49:15 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:15.818420 25467 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:15.818608 25467 heartbeater.cc:511] Master 127.11.247.62:33751 requested a full tablet report, sending...
I20260926 05:49:15.818972 25000 ts_manager.cc:194] Registered new tserver with Master: 9f38aad9c562448f84801972553b82f5 (127.11.247.3:45611)
I20260926 05:49:15.819484 25000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:51973
I20260926 05:49:15.820942 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:49:15.835561 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.823987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.11.247.62:33751)
0926 05:49:15.826673 (+ 2686us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:15.826675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:15.826684 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:15.826711 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:15.826712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:15.827421 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:15.828200 (+ 779us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:15.828263 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.829021 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.830100 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.830422 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.830839 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.832064 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:15.832065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:15.832066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:15.832074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:15.833836 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:15.834250 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:15.834251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:15.834251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:15.834317 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:15.834540 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:15.834541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:15.834541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:15.834708 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:15.835269 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:15.835313 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 05:49:15.835409 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2660,"thread_start_us":80,"threads_started":1}
I20260926 05:49:15.838361 25000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:49:15.841208 25000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:49:15.882406 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.869772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:45611 (local address 127.0.0.1:50116)
0926 05:49:15.870940 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:15.870947 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:15.870978 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:15.872690 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:15.872691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:15.872693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:15.872752 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:15.872754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.874060 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.874062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:15.875872 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:15.875876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.882040 (+ 6164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.882042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:15.882161 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.882177 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:49:15.882244 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:49:15.882306 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":111,"threads_started":1}
I20260926 05:49:15.885476 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.869073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:46569 (local address 127.0.0.1:32908)
0926 05:49:15.870531 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:15.870601 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:15.870676 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:15.871334 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:15.871338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:15.871340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:15.871426 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:15.871428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.878734 (+ 7306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.878735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:15.879816 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:15.879819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.885225 (+ 5406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.885226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:15.885307 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.885312 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:49:15.885343 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 05:49:15.885389 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"thread_start_us":93,"threads_started":1}
I20260926 05:49:15.886227 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.869512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.11.247.1:46569)
0926 05:49:15.871079 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:15.871084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:15.871095 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:15.871132 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:15.871134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:15.871151 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:15.871223 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:15.871508 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.877072 (+ 5564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.879934 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.883078 (+ 3144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.885965 (+ 2887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.886061 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:15.886091 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 05:49:15.886145 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":47,"threads_started":1}
I20260926 05:49:15.887358 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.869314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46693 (local address 127.0.0.1:53724)
0926 05:49:15.870826 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:15.870836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:15.870872 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:15.876313 (+ 5441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:15.876314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:15.876317 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:15.876368 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:15.876369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.883195 (+ 6826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.883197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:15.884366 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:15.884368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.887110 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.887111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:15.887187 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.887192 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:49:15.887238 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:49:15.887282 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"thread_start_us":66,"threads_started":1}
I20260926 05:49:15.888084 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.870085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.11.247.3:45611)
0926 05:49:15.872457 (+ 2372us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:15.872462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:15.872472 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:49:15.872506 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:15.872508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:15.872524 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:15.872599 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:15.872849 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.873960 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.881035 (+ 7075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.881861 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.881937 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.882002 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:15.887921 (+ 5919us) server_negotiation.cc:299] Negotiation successful
0926 05:49:15.888006 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2334,"thread_start_us":58,"threads_started":1}
I20260926 05:49:15.888396 25118 tablet_service.cc:1467] Processing CreateTablet for tablet 113e013bffab42f1986cea7df8e40412 (DEFAULT_TABLE table=test-table [id=7f668ce833034f3dbd0d16ecd8a9e643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:15.888686 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:15.869417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.11.247.2:46693)
0926 05:49:15.876072 (+ 6655us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:15.876078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:15.876089 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:15.876127 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:15.876129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:15.876145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:15.876223 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:15.877174 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.878620 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.884478 (+ 5858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:15.886950 (+ 2472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:15.887023 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:15.887074 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:15.888558 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0926 05:49:15.888613 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6611,"thread_start_us":58,"threads_started":1}
I20260926 05:49:15.888805 25118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113e013bffab42f1986cea7df8e40412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:15.888916 25389 tablet_service.cc:1467] Processing CreateTablet for tablet 113e013bffab42f1986cea7df8e40412 (DEFAULT_TABLE table=test-table [id=7f668ce833034f3dbd0d16ecd8a9e643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:15.889271 25389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113e013bffab42f1986cea7df8e40412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:15.892957 25248 tablet_service.cc:1467] Processing CreateTablet for tablet 113e013bffab42f1986cea7df8e40412 (DEFAULT_TABLE table=test-table [id=7f668ce833034f3dbd0d16ecd8a9e643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:15.893301 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113e013bffab42f1986cea7df8e40412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:15.896948 25502 tablet_bootstrap.cc:492] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Bootstrap starting.
I20260926 05:49:15.897713 25502 tablet_bootstrap.cc:654] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:15.898010 25502 log.cc:824] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:15.898746 25501 tablet_bootstrap.cc:492] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: Bootstrap starting.
I20260926 05:49:15.899435 25501 tablet_bootstrap.cc:654] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:15.899641 25503 tablet_bootstrap.cc:492] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Bootstrap starting.
I20260926 05:49:15.900219 25503 tablet_bootstrap.cc:654] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:15.900597 25501 log.cc:824] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:15.901413 25503 log.cc:824] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:15.902040 25502 tablet_bootstrap.cc:492] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: No bootstrap required, opened a new log
I20260926 05:49:15.902153 25502 ts_tablet_manager.cc:1397] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 05:49:15.904708 25502 raft_consensus.cc:348] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:15.904997 25502 raft_consensus.cc:374] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:15.905078 25502 raft_consensus.cc:729] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f38aad9c562448f84801972553b82f5, State: Initialized, Role: FOLLOWER
I20260926 05:49:15.905277 25502 consensus_queue.cc:260] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:15.905614 25502 ts_tablet_manager.cc:1428] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:49:15.905761 25467 heartbeater.cc:503] Master 127.11.247.62:33751 was elected leader, sending a full tablet report...
I20260926 05:49:15.908427 25501 tablet_bootstrap.cc:492] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: No bootstrap required, opened a new log
I20260926 05:49:15.908818 25503 tablet_bootstrap.cc:492] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: No bootstrap required, opened a new log
I20260926 05:49:15.908951 25501 ts_tablet_manager.cc:1397] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260926 05:49:15.911566 25501 raft_consensus.cc:348] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:15.911851 25503 ts_tablet_manager.cc:1397] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260926 05:49:15.914698 25501 raft_consensus.cc:374] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:15.914815 25501 raft_consensus.cc:729] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215f186c37e24257b0ddcc9274340c9d, State: Initialized, Role: FOLLOWER
I20260926 05:49:15.914388 25503 raft_consensus.cc:348] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:15.915509 25501 consensus_queue.cc:260] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:15.915663 25503 raft_consensus.cc:374] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:15.915740 25503 raft_consensus.cc:729] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df43c43530064d11a6afbae7e36199a4, State: Initialized, Role: FOLLOWER
I20260926 05:49:15.916028 25501 ts_tablet_manager.cc:1428] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260926 05:49:15.916215 25503 consensus_queue.cc:260] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:15.916538 25503 ts_tablet_manager.cc:1428] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 05:49:15.916689 25199 heartbeater.cc:503] Master 127.11.247.62:33751 was elected leader, sending a full tablet report...
I20260926 05:49:15.917438 25333 heartbeater.cc:503] Master 127.11.247.62:33751 was elected leader, sending a full tablet report...
W20260926 05:49:16.054725 25468 tablet.cc:2369] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:16.081281 25509 raft_consensus.cc:482] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:16.081519 25509 raft_consensus.cc:504] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:16.082007 25509 leader_election.cc:288] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 215f186c37e24257b0ddcc9274340c9d (127.11.247.1:46569), 9f38aad9c562448f84801972553b82f5 (127.11.247.3:45611)
W20260926 05:49:16.091642 25334 tablet.cc:2369] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:16.092144 25507 raft_consensus.cc:482] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:16.092265 25507 raft_consensus.cc:504] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:16.092728 25507 leader_election.cc:288] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 215f186c37e24257b0ddcc9274340c9d (127.11.247.1:46569), df43c43530064d11a6afbae7e36199a4 (127.11.247.2:46693)
I20260926 05:49:16.094729 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.082717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:34765 (local address 127.11.247.1:46569)
0926 05:49:16.082780 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:16.082782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:16.084103 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 05:49:16.091531 (+ 7428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:16.091533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:16.091548 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:16.091579 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:16.091822 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.092511 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.093824 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.094327 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.094371 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.094381 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:16.094619 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:49:16.094662 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:49:16.095145 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.082325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:46569 (local address 127.11.247.2:34765)
0926 05:49:16.084015 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:16.084028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:16.084513 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:16.091657 (+ 7144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:16.091658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:16.091660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:16.091745 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.091747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.092617 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.092617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.093718 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.093721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.094454 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.094455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.094518 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.094521 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:49:16.095002 (+ 481us) client_negotiation.cc:240] Negotiation successful
0926 05:49:16.095049 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"thread_start_us":138,"threads_started":1}
I20260926 05:49:16.095577 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113e013bffab42f1986cea7df8e40412" candidate_uuid: "df43c43530064d11a6afbae7e36199a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215f186c37e24257b0ddcc9274340c9d" is_pre_election: true
I20260926 05:49:16.095736 25153 raft_consensus.cc:2454] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df43c43530064d11a6afbae7e36199a4 in term 0.
I20260926 05:49:16.095764 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.083057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:60495 (local address 127.11.247.3:45611)
0926 05:49:16.083111 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:16.083113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:16.086622 (+ 3509us) server_negotiation.cc:407] Connection header received
0926 05:49:16.088196 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:16.088198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:16.088209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:16.088237 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:16.088572 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.089280 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.094709 (+ 5429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.095311 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.095361 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.095369 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:16.095652 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 05:49:16.095699 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:49:16.096035 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.082783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:45611 (local address 127.11.247.2:60495)
0926 05:49:16.086520 (+ 3737us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:16.086533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:16.088084 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:16.088319 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:16.088319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:16.088321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:16.088496 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.088499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.089399 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.089400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.094613 (+ 5213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.094616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.095432 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.095433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.095563 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.095567 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:49:16.095869 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 05:49:16.095911 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3611,"thread_start_us":112,"threads_started":1}
I20260926 05:49:16.096109 25222 leader_election.cc:302] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215f186c37e24257b0ddcc9274340c9d, df43c43530064d11a6afbae7e36199a4; no voters:
I20260926 05:49:16.096360 25509 raft_consensus.cc:2781] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:49:16.096419 25509 raft_consensus.cc:482] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:49:16.096441 25509 raft_consensus.cc:3037] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:16.097780 25509 raft_consensus.cc:504] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:16.097921 25509 leader_election.cc:288] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [CANDIDATE]: Term 1 election: Requested vote from peers 215f186c37e24257b0ddcc9274340c9d (127.11.247.1:46569), 9f38aad9c562448f84801972553b82f5 (127.11.247.3:45611)
I20260926 05:49:16.098397 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113e013bffab42f1986cea7df8e40412" candidate_uuid: "df43c43530064d11a6afbae7e36199a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215f186c37e24257b0ddcc9274340c9d"
I20260926 05:49:16.098482 25153 raft_consensus.cc:3037] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:16.099845 25153 raft_consensus.cc:2454] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df43c43530064d11a6afbae7e36199a4 in term 1.
I20260926 05:49:16.100087 25222 leader_election.cc:302] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215f186c37e24257b0ddcc9274340c9d, df43c43530064d11a6afbae7e36199a4; no voters:
I20260926 05:49:16.100215 25509 raft_consensus.cc:2781] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:16.100387 25509 raft_consensus.cc:686] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 1 LEADER]: Becoming Leader. State: Replica: df43c43530064d11a6afbae7e36199a4, State: Running, Role: LEADER
I20260926 05:49:16.100485 25509 consensus_queue.cc:237] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } }
I20260926 05:49:16.101359 25000 catalog_manager.cc:5495] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to df43c43530064d11a6afbae7e36199a4 (127.11.247.2). New cstate: current_term: 1 leader_uuid: "df43c43530064d11a6afbae7e36199a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df43c43530064d11a6afbae7e36199a4" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46693 } health_report { overall_health: HEALTHY } } }
I20260926 05:49:16.102214 25421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113e013bffab42f1986cea7df8e40412" candidate_uuid: "df43c43530064d11a6afbae7e36199a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f38aad9c562448f84801972553b82f5"
I20260926 05:49:16.102272 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113e013bffab42f1986cea7df8e40412" candidate_uuid: "df43c43530064d11a6afbae7e36199a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f38aad9c562448f84801972553b82f5" is_pre_election: true
I20260926 05:49:16.102324 25421 raft_consensus.cc:3037] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:16.103852 25421 raft_consensus.cc:2454] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df43c43530064d11a6afbae7e36199a4 in term 1.
I20260926 05:49:16.104413 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.093706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:54463 (local address 127.11.247.1:46569)
0926 05:49:16.094805 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:16.094807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:16.098516 (+ 3709us) server_negotiation.cc:407] Connection header received
0926 05:49:16.098722 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:16.098723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:16.098733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:16.098763 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:16.100280 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.101128 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.103445 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.104005 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.104048 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.104058 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:16.104290 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 05:49:16.104337 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":128,"threads_started":1}
I20260926 05:49:16.104671 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.093007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:46569 (local address 127.11.247.3:54463)
0926 05:49:16.098446 (+ 5439us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:16.098459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:16.098544 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:16.100092 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:16.100093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:16.100096 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:16.100175 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.100177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.101221 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.101222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.102381 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.102384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.104125 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.104126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.104195 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.104200 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:49:16.104533 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 05:49:16.104580 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5408,"thread_start_us":151,"threads_started":1}
I20260926 05:49:16.104869 25153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113e013bffab42f1986cea7df8e40412" candidate_uuid: "9f38aad9c562448f84801972553b82f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215f186c37e24257b0ddcc9274340c9d" is_pre_election: true
I20260926 05:49:16.104964 25153 raft_consensus.cc:2379] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f38aad9c562448f84801972553b82f5 in current term 1: Already voted for candidate df43c43530064d11a6afbae7e36199a4 in this term.
I20260926 05:49:16.107291 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.093956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:57417 (local address 127.11.247.2:46693)
0926 05:49:16.094010 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:16.094012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:16.102549 (+ 8537us) server_negotiation.cc:407] Connection header received
0926 05:49:16.102647 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:16.102648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:16.102659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:16.102709 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:16.103200 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.103900 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.106350 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.106901 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.106950 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.106958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:16.107173 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:49:16.107214 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:49:16.107684 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.093287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46693 (local address 127.11.247.3:57417)
0926 05:49:16.102497 (+ 9210us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:16.102507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:16.102561 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:16.102751 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:16.102752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:16.102753 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:16.102836 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.102837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.104002 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.104003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.105065 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:16.105068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.107018 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.107019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:16.107098 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.107102 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:49:16.107559 (+ 457us) client_negotiation.cc:240] Negotiation successful
0926 05:49:16.107607 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9190,"thread_start_us":112,"threads_started":1}
I20260926 05:49:16.108130 25287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113e013bffab42f1986cea7df8e40412" candidate_uuid: "9f38aad9c562448f84801972553b82f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43c43530064d11a6afbae7e36199a4" is_pre_election: true
I20260926 05:49:16.108489 25356 leader_election.cc:302] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f38aad9c562448f84801972553b82f5; no voters: 215f186c37e24257b0ddcc9274340c9d, df43c43530064d11a6afbae7e36199a4
I20260926 05:49:16.108681 25507 raft_consensus.cc:2726] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 05:49:16.118142 25200 tablet.cc:2369] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:49:16 dist-test-slave-21p4 krb5kdc[24004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for HTTP/127.11.247.20@KRBTEST.COM
I20260926 05:49:16.155396 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.152473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.11.247.2:46693)
0926 05:49:16.152522 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:16.152524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:16.152532 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:49:16.152576 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:16.152577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:16.152601 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:16.152676 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:49:16.152832 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.153386 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.154223 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:16.154496 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:16.154570 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:16.154808 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:49:16.154972 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:49:16.155064 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:16.155284 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 05:49:16.155314 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:49:16.158463 25421 raft_consensus.cc:1261] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 1 FOLLOWER]: Refusing update from remote peer df43c43530064d11a6afbae7e36199a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:16.158927 25153 raft_consensus.cc:1261] T 113e013bffab42f1986cea7df8e40412 P 215f186c37e24257b0ddcc9274340c9d [term 1 FOLLOWER]: Refusing update from remote peer df43c43530064d11a6afbae7e36199a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:16.159000 25515 consensus_queue.cc:1035] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f38aad9c562448f84801972553b82f5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 45611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:16.159353 25515 consensus_queue.cc:1035] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "215f186c37e24257b0ddcc9274340c9d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:16.164420 25522 mvcc.cc:205] Tried to move back new op lower bound from 7333485593220681728 to 7333485592988172288. Current Snapshot: MvccSnapshot[applied={T|T < 7333485593220681728}]
I20260926 05:49:16.166141 25520 mvcc.cc:205] Tried to move back new op lower bound from 7333485593220681728 to 7333485592988172288. Current Snapshot: MvccSnapshot[applied={T|T < 7333485593220681728}]
I20260926 05:49:16.172021 25000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:16.172149 25000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:16.173182 25000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dca481ce37c04761b4aeff40f0162237: Sending DeleteTablet for 3 replicas of tablet 113e013bffab42f1986cea7df8e40412
I20260926 05:49:16.173676 25118 tablet_service.cc:1514] Processing DeleteTablet for tablet 113e013bffab42f1986cea7df8e40412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:16 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:32908
I20260926 05:49:16.173933 25248 tablet_service.cc:1514] Processing DeleteTablet for tablet 113e013bffab42f1986cea7df8e40412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:16 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:53724
I20260926 05:49:16.174491 25389 tablet_service.cc:1514] Processing DeleteTablet for tablet 113e013bffab42f1986cea7df8e40412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:16 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:50116
I20260926 05:49:16.175232 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25038
I20260926 05:49:16.178920 25531 tablet_replica.cc:330] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: stopping tablet replica
I20260926 05:49:16.179068 25531 raft_consensus.cc:2229] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:16.179204 25531 raft_consensus.cc:2258] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:16.179392 25530 tablet_replica.cc:330] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: stopping tablet replica
I20260926 05:49:16.179500 25530 raft_consensus.cc:2229] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:49:16.179620 25530 raft_consensus.cc:2258] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:16.184126 25531 ts_tablet_manager.cc:1905] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:49:16.186088 24986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:46569 (error 108)
I20260926 05:49:16.186403 25531 ts_tablet_manager.cc:1918] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:16.186480 25531 log.cc:1196] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/113e013bffab42f1986cea7df8e40412
I20260926 05:49:16.186754 25531 ts_tablet_manager.cc:1939] T 113e013bffab42f1986cea7df8e40412 P 9f38aad9c562448f84801972553b82f5: Deleting consensus metadata
I20260926 05:49:16.187016 25530 ts_tablet_manager.cc:1905] T 113e013bffab42f1986cea7df8e40412 P df43c43530064d11a6afbae7e36199a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:16.187258 24987 catalog_manager.cc:4841] TS 9f38aad9c562448f84801972553b82f5 (127.11.247.3:45611): tablet 113e013bffab42f1986cea7df8e40412 (table test-table [id=7f668ce833034f3dbd0d16ecd8a9e643]) successfully deleted
I20260926 05:49:16.187518 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25203
I20260926 05:49:16.189968 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:16.189836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:46569 (local address 127.0.0.1:32920)
0926 05:49:16.189892 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:16.189931 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:46569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:49:16.190177 24986 catalog_manager.cc:4584] TS 215f186c37e24257b0ddcc9274340c9d (127.11.247.1:46569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 113e013bffab42f1986cea7df8e40412: Network error: Client connection negotiation failed: client connection to 127.11.247.1:46569: connect: Connection refused (error 111)
W20260926 05:49:16.194051 25496 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:49:16.193707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46693 (local address 127.0.0.1:53750)
0926 05:49:16.193762 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:16.193773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:16.193814 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:16.193998 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.2:46693: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:49:16.194571 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25336
I20260926 05:49:16.200495 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 24967
I20260926 05:49:16.208678 12252 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 05:49:16.553627 12252 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 05:49:16.553730 12252 mini_ranger.cc:66] Stopping Ranger...
I20260926 05:49:16.898998 12252 mini_ranger.cc:68] Stopped Ranger
2026-09-26 05:49:16.899 UTC [24034] LOG: received smart shutdown request
2026-09-26 05:49:16.903 UTC [24034] LOG: background worker "logical replication launcher" (PID 24043) exited with exit code 1
2026-09-26 05:49:16.903 UTC [24038] LOG: shutting down
2026-09-26 05:49:17.112 UTC [24034] LOG: database system is shut down
2026-09-26T05:49:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50913 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:49:17 dist-test-slave-21p4 krb5kdc[25542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:49:17 dist-test-slave-21p4 krb5kdc[25542](info): set up 2 sockets
Sep 26 05:49:17 dist-test-slave-21p4 krb5kdc[25542](info): commencing operation
krb5kdc: starting...
W20260926 05:49:18.282793 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:49:18.413530 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34201
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34201,127.11.247.61:34889,127.11.247.60:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:18.496606 25555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:18.496903 25555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:18.496948 25555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:18.498230 25555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:18.498323 25555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:18.498347 25555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:18.498368 25555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:18.498399 25555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:18.500119 25555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34201,127.11.247.61:34889,127.11.247.60:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:18.500525 25555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:18.500838 25555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:18.506258 25561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:18.506230 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:18.506294 25560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:18.506523 25555 server_base.cc:1034] running on GCE node
I20260926 05:49:18.506985 25555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:18.507400 25555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:18.508590 25555 hybrid_clock.cc:648] HybridClock initialized: now 1790401758508571 us; error 39 us; skew 500 ppm
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:18.511539 25555 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:18.512089 25555 webserver.cc:460] Webserver started at http://127.11.247.62:36691/ using document root <none> and password file <none>
I20260926 05:49:18.512362 25555 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:18.512413 25555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:18.512598 25555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:18.513751 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "ef4325ebabc540ebb1b4d36a59a91f1a"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "40be19b47aa7f8338edb267111285029"
server_key_iv: "480526d51c4745adad17e085ffd6c663"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.514153 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "ef4325ebabc540ebb1b4d36a59a91f1a"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "40be19b47aa7f8338edb267111285029"
server_key_iv: "480526d51c4745adad17e085ffd6c663"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.515740 25555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:49:18.516607 25570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:18.516819 25555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:18.516922 25555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "ef4325ebabc540ebb1b4d36a59a91f1a"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "40be19b47aa7f8338edb267111285029"
server_key_iv: "480526d51c4745adad17e085ffd6c663"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.516997 25555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:18.534168 25555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:18.536437 25555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:18.536558 25555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:18.540550 25555 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:34201
I20260926 05:49:18.540707 25622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:34201 every 8 connection(s)
I20260926 05:49:18.540998 25555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:18.542037 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:18.543871 25623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.548974 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25555
I20260926 05:49:18.549202 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:49:18.549422 12252 external_mini_cluster.cc:1428] Setting key 6a94339e508dd219a4f10c5b3b027a03
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:18.555687 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.544644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34201 (local address 127.0.0.1:52310)
0926 05:49:18.544904 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.544958 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.545520 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.545857 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.545860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.546166 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.546332 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.546337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.547526 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.547528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.548283 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.548285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.548876 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.548877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.548951 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.549654 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.549662 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.551468 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.553662 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.553670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.553673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.553904 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.554193 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.554262 (+ 69us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.554262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.554405 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.554986 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.554992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.555060 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.555179 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.555482 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":136,"thread_start_us":77,"threads_started":1}
I20260926 05:49:18.555686 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.544812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.11.247.62:34201)
0926 05:49:18.544950 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.544961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.545076 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:49:18.545711 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.545719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.545742 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.545849 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.546446 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.547397 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.548400 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.548742 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.548905 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.551577 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.551587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.551587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.551597 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.553530 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.554012 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.554016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.554018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.554115 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.554499 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.554500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.554501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.554892 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.555013 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.555213 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.555413 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":43,"threads_started":1}
WARNING: no policy specified for kudu/127.11.247.61@KRBTEST.COM; defaulting to no policy
I20260926 05:49:18.556802 25623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.557219 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.557099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59384)
0926 05:49:18.557152 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.557196 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 05:49:18.557365 25571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
I20260926 05:49:18.557762 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.557663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59388)
0926 05:49:18.557714 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.557739 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:49:18.557927 25623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:34889: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab.
Entry for principal kudu/127.11.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab.
I20260926 05:49:18.577989 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:34889
--webserver_interface=127.11.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab
--principal=kudu/127.11.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34201,127.11.247.61:34889,127.11.247.60:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:18.607184 25623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } attempt: 1
I20260926 05:49:18.607719 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.607584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59390)
0926 05:49:18.607649 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.607691 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 05:49:18.608219 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.608092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59402)
0926 05:49:18.608164 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.608199 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 05:49:18.608316 25623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:34889: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
W20260926 05:49:18.662529 25630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:18.662849 25630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:18.662895 25630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:18.664166 25630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:18.664263 25630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:18.664299 25630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:18.664327 25630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:18.664358 25630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:18.666070 25630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34201,127.11.247.61:34889,127.11.247.60:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab
--principal=kudu/127.11.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:34889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:18.666405 25630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:18.666676 25630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:18.672255 25638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:18.672407 25636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:18.672279 25635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:18.672602 25630 server_base.cc:1034] running on GCE node
I20260926 05:49:18.672909 25630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:18.673236 25630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:18.674400 25630 hybrid_clock.cc:648] HybridClock initialized: now 1790401758674385 us; error 31 us; skew 500 ppm
I20260926 05:49:18.676478 25623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } attempt: 2
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:18.676967 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.676811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59404)
0926 05:49:18.676882 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.676922 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 05:49:18.677359 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.677261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59412)
0926 05:49:18.677305 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.677334 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 05:49:18.677465 25630 init.cc:377] Logged in from keytab as kudu/127.11.247.61@KRBTEST.COM (short username kudu)
W20260926 05:49:18.677525 25623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:34889: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34889: connect: Connection refused (error 111)
I20260926 05:49:18.677997 25630 webserver.cc:460] Webserver started at http://127.11.247.61:39083/ using document root <none> and password file <none>
I20260926 05:49:18.678252 25630 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:18.678306 25630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:18.678457 25630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:18.679597 25630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data/instance:
uuid: "eee9102b957a475f9b8f36020b428547"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "ee930467a617386a77d96876589dc458"
server_key_iv: "7e746dff309fcb4e6929e53d33100fe5"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.679970 25630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal/instance:
uuid: "eee9102b957a475f9b8f36020b428547"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "ee930467a617386a77d96876589dc458"
server_key_iv: "7e746dff309fcb4e6929e53d33100fe5"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.681382 25630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:49:18.682169 25645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:18.682407 25630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:18.682514 25630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal
uuid: "eee9102b957a475f9b8f36020b428547"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "ee930467a617386a77d96876589dc458"
server_key_iv: "7e746dff309fcb4e6929e53d33100fe5"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.682610 25630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:18.696208 25630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:18.698653 25630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:18.698827 25630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:18.703002 25630 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.61:34889
I20260926 05:49:18.703042 25697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.61:34889 every 8 connection(s)
I20260926 05:49:18.703478 25630 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
I20260926 05:49:18.704442 25698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:18.706058 25698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:18.713529 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25630
I20260926 05:49:18.713629 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-1/wal/instance
I20260926 05:49:18.713841 12252 external_mini_cluster.cc:1428] Setting key c4b92e4d8c3d12405df3425c72b7ee72
I20260926 05:49:18.715566 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.706885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.11.247.62:34201)
0926 05:49:18.706945 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.706948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.707142 (+ 194us) server_negotiation.cc:407] Connection header received
0926 05:49:18.707596 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.707597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.707606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.707668 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.708442 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.708945 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.709953 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.710179 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.710238 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.712492 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.712493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.712494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.712501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.713890 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.714424 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.714425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.714426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.714489 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.714788 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.714789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.714790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.714922 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.715005 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.715408 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.715498 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:49:18.715885 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.706823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34201 (local address 127.0.0.1:52318)
0926 05:49:18.707005 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.707049 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.707515 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.707735 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.707738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.708030 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.708307 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.708315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.709040 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.709041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.709865 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.709867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.710257 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.710258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.710365 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.710924 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.710932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.712400 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.714003 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.714013 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.714015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.714316 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.714558 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.714559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.714560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.714712 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.715038 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.715047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.715244 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.715375 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.715584 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260926 05:49:18.716598 25698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.60@KRBTEST.COM; defaulting to no policy
I20260926 05:49:18.724714 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.716874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59420)
0926 05:49:18.716933 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.716945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.716989 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.717275 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.717276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.717423 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.717475 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.717476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.718386 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.718387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.718965 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.718967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.719436 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.719437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.719506 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.720063 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.720068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.721588 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.723535 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.723536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.723536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.723706 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.723902 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.723903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.723903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.723956 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.724287 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.724288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.724333 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.724439 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.724609 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:49:18.724838 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.716933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.11.247.61:34889)
0926 05:49:18.717148 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.717152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.717166 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:49:18.717182 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.717184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.717194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.717274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.717552 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.718286 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.719043 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.719349 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.719438 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.721692 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.721703 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.721703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.721710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.723451 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.723782 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.723785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.723785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.723844 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.724012 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.724012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.724013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.724185 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.724252 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.724512 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.724684 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":183,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.60@KRBTEST.COM" created.
I20260926 05:49:18.725518 25698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.725970 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.725804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57878)
0926 05:49:18.725869 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.725937 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:49:18.726150 25646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
I20260926 05:49:18.726687 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.726547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57886)
0926 05:49:18.726617 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.726652 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 05:49:18.726810 25698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:33859: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab.
Entry for principal kudu/127.11.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab.
I20260926 05:49:18.742808 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:33859
--webserver_interface=127.11.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab
--principal=kudu/127.11.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34201,127.11.247.61:34889,127.11.247.60:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:18.768725 25623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } attempt: 3
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:49:18.776041 25698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } attempt: 1
I20260926 05:49:18.776612 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.776421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57902)
0926 05:49:18.776490 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.776566 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 05:49:18.777048 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.776925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57918)
0926 05:49:18.776989 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.777021 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 05:49:18.777159 25698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:33859: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
I20260926 05:49:18.777478 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.769318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.11.247.61:34889)
0926 05:49:18.769378 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.769380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.769457 (+ 77us) server_negotiation.cc:407] Connection header received
0926 05:49:18.769515 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.769516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.769530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.769612 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.769991 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.770819 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.771674 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.772008 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.772088 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.774518 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.774520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.774520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.774529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.776162 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.776598 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.776599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.776600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.776661 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.776866 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.776868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.776868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.777034 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.777123 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.777304 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.777394 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:49:18.777491 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.769300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59428)
0926 05:49:18.769374 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.769388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.769457 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.769649 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.769650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.769833 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.769904 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.769906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.770922 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.770923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.771557 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.771560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.772115 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.772117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.772233 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.772891 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.772897 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.774418 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.776270 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.776272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.776272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.776515 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.776732 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.776733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.776733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.776794 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.777115 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.777116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.777175 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.777267 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.777392 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 05:49:18.777870 25623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.778218 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.778119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57924)
0926 05:49:18.778170 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.778199 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:49:18.778519 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.778433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57928)
0926 05:49:18.778476 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.778500 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:49:18.778630 25623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:33859: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
I20260926 05:49:18.809787 25623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } attempt: 1
I20260926 05:49:18.810397 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.810188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57930)
0926 05:49:18.810270 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.810313 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 05:49:18.810794 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.810692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57936)
0926 05:49:18.810741 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.810773 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:49:18.810930 25623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:33859: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
W20260926 05:49:18.831847 25705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:18.832125 25705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:18.832166 25705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:18.833377 25705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:18.833452 25705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:18.833479 25705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:18.833510 25705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:18.833535 25705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:18.835224 25705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34201,127.11.247.61:34889,127.11.247.60:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab
--principal=kudu/127.11.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:18.835534 25705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:18.835773 25705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:18.840896 25710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:18.840919 25711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:18.840909 25713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:18.841210 25705 server_base.cc:1034] running on GCE node
I20260926 05:49:18.841477 25705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:18.841804 25705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:18.843031 25705 hybrid_clock.cc:648] HybridClock initialized: now 1790401758842986 us; error 36 us; skew 500 ppm
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:18.845346 25698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } attempt: 2
I20260926 05:49:18.845801 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.845644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57944)
0926 05:49:18.845715 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.845754 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:49:18.845816 25705 init.cc:377] Logged in from keytab as kudu/127.11.247.60@KRBTEST.COM (short username kudu)
I20260926 05:49:18.846220 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.846112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57954)
0926 05:49:18.846163 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.846196 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 05:49:18.846367 25698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:33859: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
I20260926 05:49:18.846458 25705 webserver.cc:460] Webserver started at http://127.11.247.60:46243/ using document root <none> and password file <none>
I20260926 05:49:18.846748 25705 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:18.846828 25705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:18.846973 25705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:18.848096 25705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data/instance:
uuid: "eb7fd3e2bdb3409180566d5cc9654234"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "63e33b3ca1e08ca0d5b07b19243bc585"
server_key_iv: "444c219fdb7adbc9437e2b377b62a8bb"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.848484 25705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal/instance:
uuid: "eb7fd3e2bdb3409180566d5cc9654234"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "63e33b3ca1e08ca0d5b07b19243bc585"
server_key_iv: "444c219fdb7adbc9437e2b377b62a8bb"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.850015 25705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:49:18.850888 25720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:18.851061 25705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:18.851166 25705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal
uuid: "eb7fd3e2bdb3409180566d5cc9654234"
format_stamp: "Formatted at 2026-09-26 05:49:18 on dist-test-slave-21p4"
server_key: "63e33b3ca1e08ca0d5b07b19243bc585"
server_key_iv: "444c219fdb7adbc9437e2b377b62a8bb"
server_key_version: "encryptionkey@0"
I20260926 05:49:18.851257 25705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:18.886080 25623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } attempt: 2
I20260926 05:49:18.886552 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.886422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57958)
0926 05:49:18.886485 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.886522 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:49:18.886709 25705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:18.886978 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.886855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57974)
0926 05:49:18.886901 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.886943 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 05:49:18.887142 25623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:33859: Network error: Client connection negotiation failed: client connection to 127.11.247.60:33859: connect: Connection refused (error 111)
I20260926 05:49:18.889309 25705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:18.889489 25705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:18.893661 25705 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.60:33859
I20260926 05:49:18.893738 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.60:33859 every 8 connection(s)
I20260926 05:49:18.894095 25705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
I20260926 05:49:18.895092 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:18.896761 25773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.898516 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25705
I20260926 05:49:18.898669 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401595249435-12252-0/minicluster-data/master-2/wal/instance
I20260926 05:49:18.898872 12252 external_mini_cluster.cc:1428] Setting key 49c911168bcaa68aff9a51330e11efaf
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:49:18.908291 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.897570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.11.247.62:34201)
0926 05:49:18.897666 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.897668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.897864 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:49:18.898335 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.898336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.898346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.898399 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.899303 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.900092 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.901574 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.901874 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.901961 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.904613 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.904615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.904616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.904624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.906359 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.906996 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.906998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.906998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.907062 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.907395 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.907396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.907397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.907560 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.907635 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.908048 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.908176 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 05:49:18.908529 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.897532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34201 (local address 127.0.0.1:52332)
0926 05:49:18.897738 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.897776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.898215 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.898512 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.898516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.898926 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.899206 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.899212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.900211 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.900212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.901471 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.901474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.901967 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.901967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.902083 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.902688 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.902696 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.904523 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.906488 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.906497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.906500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.906919 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.907159 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.907160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.907160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.907299 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.907639 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.907645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.907874 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.908016 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.908195 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":20,"thread_start_us":123,"threads_started":1}
I20260926 05:49:18.909162 25773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.909289 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.899294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.11.247.60:33859)
0926 05:49:18.899441 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.899446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.899466 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:49:18.899524 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.899526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.899541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.899604 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.900300 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.901524 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.902396 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.902849 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.902924 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.905572 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.905581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.905581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.905595 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.907630 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.908120 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.908124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.908124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.908189 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.908480 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.908481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.908482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.908685 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.908887 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.909111 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.909198 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:18.916559 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.909464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34889 (local address 127.0.0.1:59432)
0926 05:49:18.909517 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.909528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.909572 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.909708 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.909709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.909900 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.909965 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.909968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.910913 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.910914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.911418 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.911419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.911795 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.911796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.911876 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.912321 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.912327 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.913616 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.915201 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.915202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.915202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.915422 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.915651 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.915652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.915652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.915718 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.916127 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.916128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.916210 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.916360 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.916485 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 05:49:18.916643 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.909539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.11.247.61:34889)
0926 05:49:18.909583 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.909585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.909594 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:18.909623 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.909624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.909634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.909698 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.910131 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.910824 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.911505 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.911712 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.911795 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.913718 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.913720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.913720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.913727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.915118 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.915516 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.915517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.915517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.915577 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.915809 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.915810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.915810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.915957 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.916038 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.916408 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.916526 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:49:18.917032 25773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:49:18.918433 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.910652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.11.247.62:34201)
0926 05:49:18.910698 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.910700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.910708 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:49:18.910728 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.910729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.910739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.910845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.911346 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.911810 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.912535 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.912730 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.912818 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.915366 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.915368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.915368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.915374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.917309 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.917636 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.917637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.917638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.917692 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.917895 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.917896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.917897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.918019 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.918221 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.918258 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.918332 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:49:18.924046 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.917373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.11.247.60:33859)
0926 05:49:18.917415 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.917417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.917454 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:49:18.917523 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.917524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.917533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.917586 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.917841 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.918318 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.919157 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.919461 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.919524 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.921241 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.921243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.921243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.921250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.922795 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.923151 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.923152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.923152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.923216 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.923450 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.923451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.923452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.923596 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.923664 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.923885 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.923978 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:49:18.924175 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.917296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:57998)
0926 05:49:18.917410 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.917421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.917470 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.917589 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.917590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.917720 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.917748 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.917749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.918466 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.918468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.919083 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.919085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.919522 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.919523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.919593 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.919945 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.919956 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.921145 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.922899 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.922900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.922900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.923070 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.923325 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.923326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.923326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.923381 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.923689 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.923690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.923745 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.923867 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.924075 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":85}
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:49:18.925628 25773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: Bootstrap starting.
I20260926 05:49:18.926772 25773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:18.927073 25773 log.cc:824] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:18.927889 25773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: No bootstrap required, opened a new log
I20260926 05:49:18.927930 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.919327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.11.247.61:34889)
0926 05:49:18.919362 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.919364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.919371 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:49:18.919416 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.919417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.919427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.919490 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.920169 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.920627 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.921703 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.921991 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.922061 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.924879 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.924880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.924881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.924888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.926705 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.927100 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.927101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.927102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.927162 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.927383 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.927384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.927384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.927510 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.927718 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.927755 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.927833 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:49:18.929430 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:49:18.929968 25773 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:18.930096 25773 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:18.930119 25773 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7fd3e2bdb3409180566d5cc9654234, State: Initialized, Role: FOLLOWER
I20260926 05:49:18.930270 25773 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:18.930687 25773 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:18.930701 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:18.930881 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:49:18.932690 25797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:49:18.933189 25797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:49:18.933490 25705 master_runner.cc:186] Error getting master registration for 127.11.247.62:34201: OK, instance_id { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" instance_seqno: 1790401758539337 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:49:18.936645 25705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:49:18.937537 25698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } attempt: 3
I20260926 05:49:18.939383 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.930885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.11.247.62:34201)
0926 05:49:18.930931 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.930933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.931405 (+ 472us) server_negotiation.cc:407] Connection header received
0926 05:49:18.931481 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.931483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.931495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.931550 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.932343 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.933041 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.934165 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.934435 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.934481 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.936189 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.936191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.936191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.936198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.938037 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.938458 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.938460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.938460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.938521 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.938766 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.938766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.938767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.938915 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.938995 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.939207 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.939286 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:49:18.941910 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.931179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.11.247.61:34889)
0926 05:49:18.931220 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.931222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.931229 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:49:18.931248 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.931248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.931257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.931329 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.932110 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.932801 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.934306 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.934664 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.934706 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.937979 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.937980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.937981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.937986 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.939914 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.940334 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.940335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.940336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.940399 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.940623 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.940624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.940625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.940767 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.941368 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.941716 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.941812 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:49:18.943328 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.931071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.11.247.60:33859)
0926 05:49:18.931107 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.931108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.934254 (+ 3146us) server_negotiation.cc:407] Connection header received
0926 05:49:18.935093 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.935094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.935104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.935136 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.936656 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.937217 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.938631 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.938947 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.938997 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.940563 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.940564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.940565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.940571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.942065 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.942490 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.942491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.942491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.942546 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.942796 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.942797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.942797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.942937 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.943040 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.943151 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.943231 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:49:18.950033 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.937781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:58014)
0926 05:49:18.937830 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.937841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.937879 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.942861 (+ 4982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.942862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.943019 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.943067 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.943069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.944014 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.944015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.944529 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.944531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.945039 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.945040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.945101 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.945600 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.945606 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.946869 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.948996 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.948997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.948997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.949161 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.949400 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.949401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.949401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.949502 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.949787 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.949788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.949828 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.949880 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.949963 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:49:18.950106 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.937826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.11.247.60:33859)
0926 05:49:18.942701 (+ 4875us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.942703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.942712 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:18.942733 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.942734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.942745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.943141 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.943174 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.943940 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.944616 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.944953 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.945027 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.946968 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.946969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.946970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.946977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.948911 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.949234 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.949236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.949236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.949302 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.949573 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.949574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.949574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.949722 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.949799 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.949903 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.949986 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4851,"thread_start_us":101,"threads_started":1}
I20260926 05:49:18.951643 25698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547: Bootstrap starting.
I20260926 05:49:18.952291 25698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:18.952652 25698 log.cc:824] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:18.953574 25698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547: No bootstrap required, opened a new log
I20260926 05:49:18.955250 25698 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:18.955397 25698 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:18.955443 25698 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee9102b957a475f9b8f36020b428547, State: Initialized, Role: FOLLOWER
I20260926 05:49:18.955575 25698 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:18.955891 25801 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:18.955967 25698 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:18.956012 25801 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:49:18.958386 25812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:49:18.958503 25812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:49:18.958276 25630 master_runner.cc:186] Error getting master registration for 127.11.247.62:34201: OK, instance_id { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" instance_seqno: 1790401758539337 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:49:18.961468 25630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:49:18.986291 25623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } attempt: 3
Sep 26 05:49:18 dist-test-slave-21p4 krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:49:18.995111 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.986679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:33859 (local address 127.0.0.1:58028)
0926 05:49:18.986739 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:18.986751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:18.986788 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:18.986924 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:18.986925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:18.987122 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:18.987180 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.987183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.988185 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.988186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.989017 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:18.989020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.989561 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.989562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:18.989625 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.990166 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:18.990172 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:18.991869 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:18.993864 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.993865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.993866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.994118 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.994351 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:18.994352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:18.994352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:18.994420 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:18.994746 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:18.994747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:18.994807 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 05:49:18.994884 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:49:18.994999 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:49:18.995111 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:18.986736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.11.247.60:33859)
0926 05:49:18.986778 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:18.986781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:18.986790 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:18.986847 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:18.986847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:18.986857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:18.986920 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:18.987267 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.988088 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.989114 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:18.989474 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:18.989553 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:18.991972 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:18.991974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:18.991974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:18.991983 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:18.993759 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.994206 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.994207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.994207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.994274 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:18.994494 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:18.994495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:18.994495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:18.994673 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:18.994743 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:18.994925 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 05:49:18.995015 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:49:18.997498 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: Bootstrap starting.
I20260926 05:49:18.998277 25623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:18.998621 25623 log.cc:824] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:18.999570 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: No bootstrap required, opened a new log
I20260926 05:49:19.001674 25623 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:19.001840 25623 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:19.001875 25623 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4325ebabc540ebb1b4d36a59a91f1a, State: Initialized, Role: FOLLOWER
I20260926 05:49:19.002048 25623 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:19.002353 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.002408 25623 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:19.002492 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:49:19.004480 25826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:49:19.004611 25826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:49:19.006865 25555 master_runner.cc:263] No leader master found from master ef4325ebabc540ebb1b4d36a59a91f1a
I20260926 05:49:19.006937 25555 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 05:49:19.037384 25576 tablet.cc:2369] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:19.068686 25801 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:19.068825 25801 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:19.069073 25801 leader_election.cc:288] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef4325ebabc540ebb1b4d36a59a91f1a (127.11.247.62:34201), eb7fd3e2bdb3409180566d5cc9654234 (127.11.247.60:33859)
I20260926 05:49:19.069475 25747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee9102b957a475f9b8f36020b428547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7fd3e2bdb3409180566d5cc9654234" is_pre_election: true
I20260926 05:49:19.069636 25747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee9102b957a475f9b8f36020b428547 in term 0.
I20260926 05:49:19.070240 25597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee9102b957a475f9b8f36020b428547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" is_pre_election: true
I20260926 05:49:19.070384 25597 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee9102b957a475f9b8f36020b428547 in term 0.
I20260926 05:49:19.070672 25646 leader_election.cc:302] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb7fd3e2bdb3409180566d5cc9654234, eee9102b957a475f9b8f36020b428547; no voters:
I20260926 05:49:19.071050 25801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:49:19.071112 25801 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:49:19.071134 25801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:19.072073 25801 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:19.072207 25801 leader_election.cc:288] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [CANDIDATE]: Term 1 election: Requested vote from peers ef4325ebabc540ebb1b4d36a59a91f1a (127.11.247.62:34201), eb7fd3e2bdb3409180566d5cc9654234 (127.11.247.60:33859)
I20260926 05:49:19.072685 25597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee9102b957a475f9b8f36020b428547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a"
I20260926 05:49:19.073057 25747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee9102b957a475f9b8f36020b428547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7fd3e2bdb3409180566d5cc9654234"
I20260926 05:49:19.073148 25747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:19.074195 25747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee9102b957a475f9b8f36020b428547 in term 1.
I20260926 05:49:19.074420 25646 leader_election.cc:302] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb7fd3e2bdb3409180566d5cc9654234, eee9102b957a475f9b8f36020b428547; no voters:
I20260926 05:49:19.072809 25597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:19.075680 25597 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee9102b957a475f9b8f36020b428547 in term 1.
I20260926 05:49:19.077019 25801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:19.077185 25801 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [term 1 LEADER]: Becoming Leader. State: Replica: eee9102b957a475f9b8f36020b428547, State: Running, Role: LEADER
I20260926 05:49:19.077283 25801 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } }
I20260926 05:49:19.077714 25801 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee9102b957a475f9b8f36020b428547. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.077783 25801 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:19.078125 25830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:19.078351 25830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:19.080659 25747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [term 1 FOLLOWER]: Refusing update from remote peer eee9102b957a475f9b8f36020b428547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:19.081106 25801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:19.082607 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee9102b957a475f9b8f36020b428547. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.082715 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:49:19.083288 25597 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [term 1 FOLLOWER]: Refusing update from remote peer eee9102b957a475f9b8f36020b428547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:19.083631 25801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:19.085978 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee9102b957a475f9b8f36020b428547. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.086091 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:49:19.087711 25828 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.087826 25828 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:19.088441 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.088546 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:49:19.088819 25831 mvcc.cc:205] Tried to move back new op lower bound from 7333485605191643136 to 7333485605181149184. Current Snapshot: MvccSnapshot[applied={T|T < 7333485605191643136}]
I20260926 05:49:19.088950 25834 mvcc.cc:205] Tried to move back new op lower bound from 7333485605191643136 to 7333485605181149184. Current Snapshot: MvccSnapshot[applied={T|T < 7333485605191643136}]
I20260926 05:49:19.089951 25801 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.090036 25801 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:19.090487 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eee9102b957a475f9b8f36020b428547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef4325ebabc540ebb1b4d36a59a91f1a" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34201 } } peers { permanent_uuid: "eee9102b957a475f9b8f36020b428547" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34889 } } peers { permanent_uuid: "eb7fd3e2bdb3409180566d5cc9654234" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 33859 } } }
I20260926 05:49:19.090600 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:49:19.091341 25832 mvcc.cc:205] Tried to move back new op lower bound from 7333485605191643136 to 7333485605181149184. Current Snapshot: MvccSnapshot[applied={T|T < 7333485605191643136}]
I20260926 05:49:19.092338 25830 catalog_manager.cc:1347] Generated new cluster ID: 5c9e21c29a144735b257728ce28523c1
I20260926 05:49:19.092383 25830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:49:19.105050 25830 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:49:19.105937 25830 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:19.115545 25830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eee9102b957a475f9b8f36020b428547: Generated new TSK 0
I20260926 05:49:19.115880 25830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:49:19.933710 25797 catalog_manager.cc:1259] Loaded cluster ID: 5c9e21c29a144735b257728ce28523c1
I20260926 05:49:19.933818 25797 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: loading cluster ID for follower catalog manager: success
I20260926 05:49:19.934685 25797 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: acquiring CA information for follower catalog manager: success
I20260926 05:49:19.935045 25797 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb7fd3e2bdb3409180566d5cc9654234: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:49:20.005137 25826 catalog_manager.cc:1259] Loaded cluster ID: 5c9e21c29a144735b257728ce28523c1
I20260926 05:49:20.005219 25826 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: loading cluster ID for follower catalog manager: success
I20260926 05:49:20.006156 25826 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: acquiring CA information for follower catalog manager: success
I20260926 05:49:20.006493 25826 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef4325ebabc540ebb1b4d36a59a91f1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:49:20.171006 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25555
I20260926 05:49:20.176424 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25630
I20260926 05:49:20.181834 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25705
2026-09-26T05:49:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2952 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:49:20 dist-test-slave-21p4 krb5kdc[25846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:49:20 dist-test-slave-21p4 krb5kdc[25846](info): set up 2 sockets
Sep 26 05:49:20 dist-test-slave-21p4 krb5kdc[25846](info): commencing operation
krb5kdc: starting...
W20260926 05:49:21.226732 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:49:21 dist-test-slave-21p4 krb5kdc[25846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:49:21.349855 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34601
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:21.432152 25859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:21.432415 25859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:21.432464 25859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:21.433632 25859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:21.433704 25859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:21.433730 25859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:21.433761 25859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:21.433785 25859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:21.435400 25859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:21.435695 25859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:21.435940 25859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:21.440762 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:21.440975 25867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:21.440991 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:21.441262 25859 server_base.cc:1034] running on GCE node
I20260926 05:49:21.441515 25859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:21.441826 25859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:21.442991 25859 hybrid_clock.cc:648] HybridClock initialized: now 1790401761442976 us; error 45 us; skew 500 ppm
Sep 26 05:49:21 dist-test-slave-21p4 krb5kdc[25846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401761, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:21.445793 25859 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:21.446292 25859 webserver.cc:460] Webserver started at http://127.11.247.62:38031/ using document root <none> and password file <none>
I20260926 05:49:21.446646 25859 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:21.446743 25859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:21.446955 25859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:21.448103 25859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "863186e56b144cb3a9393eed32b66a9f"
format_stamp: "Formatted at 2026-09-26 05:49:21 on dist-test-slave-21p4"
server_key: "fc2b55a6dfe1a4331e5b2631c9dbab04"
server_key_iv: "45a20cf4f1e8869e5d4c42f8fe031b25"
server_key_version: "encryptionkey@0"
I20260926 05:49:21.448473 25859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "863186e56b144cb3a9393eed32b66a9f"
format_stamp: "Formatted at 2026-09-26 05:49:21 on dist-test-slave-21p4"
server_key: "fc2b55a6dfe1a4331e5b2631c9dbab04"
server_key_iv: "45a20cf4f1e8869e5d4c42f8fe031b25"
server_key_version: "encryptionkey@0"
I20260926 05:49:21.449878 25859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:49:21.450718 25874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:21.450904 25859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:21.451021 25859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "863186e56b144cb3a9393eed32b66a9f"
format_stamp: "Formatted at 2026-09-26 05:49:21 on dist-test-slave-21p4"
server_key: "fc2b55a6dfe1a4331e5b2631c9dbab04"
server_key_iv: "45a20cf4f1e8869e5d4c42f8fe031b25"
server_key_version: "encryptionkey@0"
I20260926 05:49:21.451110 25859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:21.464231 25859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:21.466889 25859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:21.467079 25859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:21.471123 25859 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:34601
I20260926 05:49:21.471159 25926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:34601 every 8 connection(s)
I20260926 05:49:21.471526 25859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:21.472462 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:21.474982 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25859
I20260926 05:49:21.475088 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:49:21.475152 25927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f: Bootstrap starting.
I20260926 05:49:21.475301 12252 external_mini_cluster.cc:1428] Setting key d6017f8cf5cb8e1934710c1be3f1812e
I20260926 05:49:21.475927 25927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:21.476325 25927 log.cc:824] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:21.477169 25927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f: No bootstrap required, opened a new log
I20260926 05:49:21.479516 25927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863186e56b144cb3a9393eed32b66a9f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34601 } }
I20260926 05:49:21.479703 25927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:21.479748 25927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863186e56b144cb3a9393eed32b66a9f, State: Initialized, Role: FOLLOWER
I20260926 05:49:21.479921 25927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863186e56b144cb3a9393eed32b66a9f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34601 } }
I20260926 05:49:21.480057 25927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:21.480118 25927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:21.480171 25927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:21.480974 25927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863186e56b144cb3a9393eed32b66a9f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34601 } }
I20260926 05:49:21.481097 25927 leader_election.cc:302] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 863186e56b144cb3a9393eed32b66a9f; no voters:
I20260926 05:49:21.481277 25927 leader_election.cc:288] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:49:21.481326 25932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:21.481546 25932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [term 1 LEADER]: Becoming Leader. State: Replica: 863186e56b144cb3a9393eed32b66a9f, State: Running, Role: LEADER
I20260926 05:49:21.481566 25927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:21.481647 25932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863186e56b144cb3a9393eed32b66a9f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34601 } }
I20260926 05:49:21.482076 25933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "863186e56b144cb3a9393eed32b66a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863186e56b144cb3a9393eed32b66a9f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34601 } } }
I20260926 05:49:21.482155 25933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:21.482064 25932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 863186e56b144cb3a9393eed32b66a9f. Latest consensus state: current_term: 1 leader_uuid: "863186e56b144cb3a9393eed32b66a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863186e56b144cb3a9393eed32b66a9f" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34601 } } }
I20260926 05:49:21.482290 25932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f [sys.catalog]: This master's current role is: LEADER
Sep 26 05:49:21 dist-test-slave-21p4 krb5kdc[25846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
W20260926 05:49:21.483574 25948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:49:21.483634 25948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:49:21.483691 25940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:21.483867 25940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:21.485471 25940 catalog_manager.cc:1347] Generated new cluster ID: 1d9fa89f5f4c41658a9f8d29d8872def
I20260926 05:49:21.485532 25940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:49:21.487244 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:21.475869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.11.247.62:34601)
0926 05:49:21.476074 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:21.476084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:21.476120 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:49:21.476653 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:21.476661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:21.476716 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:21.476844 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:21.477699 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:21.478664 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:21.479785 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:21.480399 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:21.480489 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:21.483522 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:21.483527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:21.483527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:21.483539 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:21.485537 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:21.485965 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:21.485968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:21.485968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:21.486068 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:21.486294 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:21.486295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:21.486295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:21.486615 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:21.486757 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:21.486880 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 05:49:21.487018 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260926 05:49:21.488257 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:49:21.493287 25940 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:49:21.493784 25940 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:21.498749 25940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 863186e56b144cb3a9393eed32b66a9f: Generated new TSK 0
I20260926 05:49:21.499019 25940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:49:21.542388 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25859
2026-09-26T05:49:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1357 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:49:21 dist-test-slave-21p4 krb5kdc[25954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:49:21 dist-test-slave-21p4 krb5kdc[25954](info): set up 2 sockets
Sep 26 05:49:21 dist-test-slave-21p4 krb5kdc[25954](info): commencing operation
krb5kdc: starting...
W20260926 05:49:22.587016 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:49:22 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:49:22.709532 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42097
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:22.792057 25967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:22.792311 25967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:22.792352 25967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:22.793529 25967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:22.793604 25967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:22.793632 25967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:22.793663 25967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:22.793687 25967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:22.795239 25967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:22.795529 25967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:22.795770 25967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:22.800776 25973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:22.800758 25972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:22.800756 25975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:22.801060 25967 server_base.cc:1034] running on GCE node
I20260926 05:49:22.801307 25967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:22.801625 25967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:22.802796 25967 hybrid_clock.cc:648] HybridClock initialized: now 1790401762802773 us; error 36 us; skew 500 ppm
Sep 26 05:49:22 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401762, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:22.811985 25967 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:22.812525 25967 webserver.cc:460] Webserver started at http://127.11.247.62:34341/ using document root <none> and password file <none>
I20260926 05:49:22.812803 25967 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:22.812891 25967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:22.813046 25967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:22.814225 25967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "d7de304b39c94acf8c695ea5848d8ca1"
format_stamp: "Formatted at 2026-09-26 05:49:22 on dist-test-slave-21p4"
server_key: "1269e7004c9cb86cf70e72bf972c4dcc"
server_key_iv: "1214bc94430068d6a5de8aca2aa49f12"
server_key_version: "encryptionkey@0"
I20260926 05:49:22.814683 25967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "d7de304b39c94acf8c695ea5848d8ca1"
format_stamp: "Formatted at 2026-09-26 05:49:22 on dist-test-slave-21p4"
server_key: "1269e7004c9cb86cf70e72bf972c4dcc"
server_key_iv: "1214bc94430068d6a5de8aca2aa49f12"
server_key_version: "encryptionkey@0"
I20260926 05:49:22.816202 25967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:22.817076 25982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:22.817282 25967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:22.817381 25967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "d7de304b39c94acf8c695ea5848d8ca1"
format_stamp: "Formatted at 2026-09-26 05:49:22 on dist-test-slave-21p4"
server_key: "1269e7004c9cb86cf70e72bf972c4dcc"
server_key_iv: "1214bc94430068d6a5de8aca2aa49f12"
server_key_version: "encryptionkey@0"
I20260926 05:49:22.817479 25967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:22.839529 25967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:22.841701 25967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:22.841874 25967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:22.846141 25967 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:42097
I20260926 05:49:22.846176 26034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:42097 every 8 connection(s)
I20260926 05:49:22.846562 25967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:22.847689 26035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:22.850178 26035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap starting.
I20260926 05:49:22.850867 26035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:22.851161 26035 log.cc:824] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:22.851989 26035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: No bootstrap required, opened a new log
I20260926 05:49:22.853750 26035 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:22.853914 26035 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:22.853940 26035 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7de304b39c94acf8c695ea5848d8ca1, State: Initialized, Role: FOLLOWER
I20260926 05:49:22.854141 26035 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:22.854225 26035 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:22.854270 26035 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:22.854323 26035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:22.854808 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 25967
I20260926 05:49:22.854930 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:49:22.855152 12252 external_mini_cluster.cc:1428] Setting key 3843cd2a66b69246dd245895bd0667e6
I20260926 05:49:22.855536 26035 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:22.855746 26035 leader_election.cc:302] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7de304b39c94acf8c695ea5848d8ca1; no voters:
I20260926 05:49:22.855955 26035 leader_election.cc:288] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:49:22.856035 26040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:22.856221 26035 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:22.856223 26040 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 LEADER]: Becoming Leader. State: Replica: d7de304b39c94acf8c695ea5848d8ca1, State: Running, Role: LEADER
I20260926 05:49:22.857666 26040 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:22.858343 26041 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7de304b39c94acf8c695ea5848d8ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } } }
I20260926 05:49:22.858460 26042 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7de304b39c94acf8c695ea5848d8ca1. Latest consensus state: current_term: 1 leader_uuid: "d7de304b39c94acf8c695ea5848d8ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } } }
I20260926 05:49:22.859122 26041 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:22.859133 26042 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: This master's current role is: LEADER
W20260926 05:49:22.858856 26054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:49:22.859352 26054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:49:22.859424 26057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:22.859596 26057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:22.861310 26057 catalog_manager.cc:1347] Generated new cluster ID: aa50ff9651c747b38f5f85bcc9441333
I20260926 05:49:22.861368 26057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:49:22 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:22.866225 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:22.855691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.11.247.62:42097)
0926 05:49:22.855946 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:22.855958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:22.855997 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:49:22.856134 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:22.856141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:22.856203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:22.856361 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:22.857010 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:22.858242 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:22.859395 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:22.860024 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:22.860139 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:22.862821 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:22.862825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:22.862825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:22.862837 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:22.864611 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:22.865002 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:22.865005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:22.865006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:22.865113 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:22.865342 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:22.865343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:22.865344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:22.865601 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:22.865786 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:22.865868 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 05:49:22.866017 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260926 05:49:22.872713 26057 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:49:22.873466 26057 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:22.877682 26057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Generated new TSK 0
I20260926 05:49:22.877923 26057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:49:22.909559 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42097
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:22.999056 26061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:22.999270 26061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:22.999297 26061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:23.000533 26061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:23.000581 26061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:23.000610 26061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:49:23.002138 26061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:23.002399 26061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:23.002719 26061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:23.008132 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.008132 26061 server_base.cc:1034] running on GCE node
W20260926 05:49:23.008167 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.008157 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.008539 26061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:23.008832 26061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:23.010013 26061 hybrid_clock.cc:648] HybridClock initialized: now 1790401763009998 us; error 34 us; skew 500 ppm
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:23.012912 26061 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:49:23.013350 26061 webserver.cc:460] Webserver started at http://127.11.247.1:38907/ using document root <none> and password file <none>
I20260926 05:49:23.013571 26061 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:23.013613 26061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:23.013705 26061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:23.014847 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "30b074e08e0445b7be845fd2a1159f74"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "b8322ba523f0debf330ea9f43f7f1ba9"
server_key_iv: "4b07b4b364f3d2c8116829e311069bc4"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.015224 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "30b074e08e0445b7be845fd2a1159f74"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "b8322ba523f0debf330ea9f43f7f1ba9"
server_key_iv: "4b07b4b364f3d2c8116829e311069bc4"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.016594 26061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:49:23.017349 26076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.017601 26061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:23.017688 26061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "30b074e08e0445b7be845fd2a1159f74"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "b8322ba523f0debf330ea9f43f7f1ba9"
server_key_iv: "4b07b4b364f3d2c8116829e311069bc4"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.017750 26061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:23.028743 26061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:23.030824 26061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:23.030944 26061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:23.031152 26061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:23.031423 26061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:23.031452 26061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.031476 26061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:23.031488 26061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.037518 26061 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:40061
I20260926 05:49:23.037621 26189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:40061 every 8 connection(s)
I20260926 05:49:23.037949 26061 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:23.045539 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26061
I20260926 05:49:23.045657 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:49:23.045902 12252 external_mini_cluster.cc:1428] Setting key 9218018f09daf495192483de15553183
I20260926 05:49:23.047959 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.038837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:46323 (local address 127.11.247.62:42097)
0926 05:49:23.038927 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.038930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.039091 (+ 161us) server_negotiation.cc:407] Connection header received
0926 05:49:23.039647 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.039649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.039662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.039733 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:23.040728 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.041240 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.042297 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.042506 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.042565 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.044739 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:23.044740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:23.044741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:23.044748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:23.046433 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.046931 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.046933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.046933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.046999 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.047272 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.047272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.047273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.047399 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:23.047477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.047759 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.047862 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 05:49:23.048244 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.038774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:46323)
0926 05:49:23.038953 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.038992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.039507 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.039791 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.039794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.040389 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:23.040604 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.040609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.041338 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.041339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.042200 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.042202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.042619 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.042620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.042728 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.043218 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:23.043239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:23.044646 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:23.046534 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.046544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.046547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.046849 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.047080 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.047081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.047081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.047195 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.047468 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:23.047475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:23.047623 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.047738 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.047916 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260926 05:49:23.048745 26190 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42097
I20260926 05:49:23.048874 26190 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:23.049088 26190 heartbeater.cc:511] Master 127.11.247.62:42097 requested a full tablet report, sending...
I20260926 05:49:23.049754 25999 ts_manager.cc:194] Registered new tserver with Master: 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1:40061)
I20260926 05:49:23.050786 25999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:46323
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:49:23.073930 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42097
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:23.160765 26195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:23.161024 26195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:23.161054 26195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:23.162694 26195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:23.162770 26195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:23.162804 26195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:49:23.164681 26195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:23.165043 26195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:23.165323 26195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:23.170946 26200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.170946 26201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.170989 26195 server_base.cc:1034] running on GCE node
W20260926 05:49:23.170979 26203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.171528 26195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:23.171823 26195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:23.173030 26195 hybrid_clock.cc:648] HybridClock initialized: now 1790401763173017 us; error 30 us; skew 500 ppm
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:23.176057 26195 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:49:23.176570 26195 webserver.cc:460] Webserver started at http://127.11.247.2:41575/ using document root <none> and password file <none>
I20260926 05:49:23.176839 26195 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:23.176914 26195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:23.177073 26195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:23.178289 26195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "fa16fa2830964094abe6ae33a15015f5"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "2ec7871d6dae41b58ec86f3eed5938e4"
server_key_iv: "de818e98a849c55cb4331275c948de82"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.178771 26195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "fa16fa2830964094abe6ae33a15015f5"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "2ec7871d6dae41b58ec86f3eed5938e4"
server_key_iv: "de818e98a849c55cb4331275c948de82"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.180301 26195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:23.181378 26210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.181747 26195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:23.181885 26195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "fa16fa2830964094abe6ae33a15015f5"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "2ec7871d6dae41b58ec86f3eed5938e4"
server_key_iv: "de818e98a849c55cb4331275c948de82"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.181975 26195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:23.195632 26195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:23.198007 26195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:23.198179 26195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:23.198443 26195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:23.198910 26195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:23.198971 26195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.199021 26195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:23.199051 26195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.205971 26195 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35675
I20260926 05:49:23.206413 26195 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:49:23.207541 26323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35675 every 8 connection(s)
I20260926 05:49:23.209599 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26195
I20260926 05:49:23.209722 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:49:23.209908 12252 external_mini_cluster.cc:1428] Setting key 04edad3747846b9fa4e24514c77312ce
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:23.217836 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.207236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:39981 (local address 127.11.247.62:42097)
0926 05:49:23.207288 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.207290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.207667 (+ 377us) server_negotiation.cc:407] Connection header received
0926 05:49:23.208268 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.208270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.208285 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.208332 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:23.209395 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.210049 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.211762 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.211985 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.212048 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.214553 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:23.214555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:23.214555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:23.214565 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:23.216193 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.216738 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.216740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.216740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.216808 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.217100 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.217102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.217102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.217237 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:23.217315 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.217631 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.217736 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:49:23.218040 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.207253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:39981)
0926 05:49:23.207530 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.207580 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.208135 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.208442 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.208446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.209051 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:23.209262 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.209267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.210201 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.210202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.211651 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.211654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.212080 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.212082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.212247 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.212886 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:23.212895 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:23.214410 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:23.216318 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.216328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.216331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.216640 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.216895 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.216896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.216896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.217022 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.217305 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:23.217312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:23.217483 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.217597 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.217781 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:49:23.218636 26324 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42097
I20260926 05:49:23.218753 26324 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:23.218997 26324 heartbeater.cc:511] Master 127.11.247.62:42097 requested a full tablet report, sending...
I20260926 05:49:23.219511 25999 ts_manager.cc:194] Registered new tserver with Master: fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675)
I20260926 05:49:23.219975 25999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:39981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:49:23.238901 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42097
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:23.324257 26329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:23.324472 26329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:23.324497 26329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:23.325659 26329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:23.325697 26329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:23.325724 26329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:49:23.327240 26329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:23.327455 26329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:23.327646 26329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:23.333112 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.333112 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.333115 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.333463 26329 server_base.cc:1034] running on GCE node
I20260926 05:49:23.333636 26329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:23.333901 26329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:23.335071 26329 hybrid_clock.cc:648] HybridClock initialized: now 1790401763335061 us; error 28 us; skew 500 ppm
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:23.337664 26329 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:49:23.338181 26329 webserver.cc:460] Webserver started at http://127.11.247.3:41325/ using document root <none> and password file <none>
I20260926 05:49:23.338433 26329 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:23.338506 26329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:23.338718 26329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:23.339901 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "8448dbaca7a84251928d864c13a86bff"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "c24476daa21f8f176ff08c4eec443306"
server_key_iv: "a2cf18b8c55c32b937998e2693707add"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.340282 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "8448dbaca7a84251928d864c13a86bff"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "c24476daa21f8f176ff08c4eec443306"
server_key_iv: "a2cf18b8c55c32b937998e2693707add"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.341787 26329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:49:23.342767 26344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.343012 26329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:23.343122 26329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "8448dbaca7a84251928d864c13a86bff"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "c24476daa21f8f176ff08c4eec443306"
server_key_iv: "a2cf18b8c55c32b937998e2693707add"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.343209 26329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:23.355134 26329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:23.357791 26329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:23.357967 26329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:23.358229 26329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:23.358563 26329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:23.358711 26329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.358759 26329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:23.358800 26329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.365345 26329 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:44477
I20260926 05:49:23.365391 26457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:44477 every 8 connection(s)
I20260926 05:49:23.365785 26329 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:23.374770 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26329
I20260926 05:49:23.374879 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:49:23.375046 12252 external_mini_cluster.cc:1428] Setting key e86e5cf08835a53d45daa664c66e192c
I20260926 05:49:23.375846 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.366756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:56701 (local address 127.11.247.62:42097)
0926 05:49:23.366832 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.366836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.367055 (+ 219us) server_negotiation.cc:407] Connection header received
0926 05:49:23.367566 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.367567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.367579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.367648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:23.368637 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.369153 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.370155 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.370397 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.370456 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.372709 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:23.372711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:23.372711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:23.372719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:23.374261 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.374724 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.374726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.374726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.374802 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.375153 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.375155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.375155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.375288 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:23.375374 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.375664 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.375753 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:49:23.376068 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.366707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:56701)
0926 05:49:23.366898 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.366952 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.367470 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.367710 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.367713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.368324 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:23.368537 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.368541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.369240 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.369241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.370068 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.370071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.370481 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.370482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.370655 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.371164 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:23.371173 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:23.372598 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:23.374346 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.374352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.374354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.374646 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.374942 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.374943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.374943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.375067 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.375369 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:23.375376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:23.375522 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.375628 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.375800 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260926 05:49:23.376626 26458 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42097
I20260926 05:49:23.376811 26458 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:23.377024 26458 heartbeater.cc:511] Master 127.11.247.62:42097 requested a full tablet report, sending...
I20260926 05:49:23.377612 25999 ts_manager.cc:194] Registered new tserver with Master: 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
I20260926 05:49:23.378098 25999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:56701
I20260926 05:49:23.387187 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:49:23.395226 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.388672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.11.247.62:42097)
0926 05:49:23.388767 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.388769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.388846 (+ 77us) server_negotiation.cc:407] Connection header received
0926 05:49:23.388931 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.388932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.388943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.389016 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:23.389663 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.390178 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.391006 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.391216 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.391295 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.392606 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:23.392607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:23.392608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:23.392616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:23.393907 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.394286 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.394287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.394288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.394351 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.394570 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.394571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.394571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.394783 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:23.394905 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.395017 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.395113 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:49:23.397274 25999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:49:23.399026 25999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:49:23.406340 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.401867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.11.247.3:44477)
0926 05:49:23.402632 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.402637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.402647 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:49:23.402683 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.402685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.402698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.402805 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:23.402929 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.404026 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.405319 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.405883 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.405984 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.406054 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.406210 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.406251 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":97,"threads_started":1}
I20260926 05:49:23.406378 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.401973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:40061 (local address 127.0.0.1:57526)
0926 05:49:23.402405 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.402415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.402465 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.402675 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.402676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.402678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:23.402741 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.402743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.404170 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.404171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.405026 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.405053 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.406033 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.406034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.406111 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.406116 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.406149 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.406293 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":99,"thread_start_us":78,"threads_started":1}
I20260926 05:49:23.406409 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.402206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.11.247.1:40061)
0926 05:49:23.402386 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.402391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.402479 (+ 88us) server_negotiation.cc:407] Connection header received
0926 05:49:23.402530 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.402532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.402549 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.402819 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:23.402894 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.404044 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.405156 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.405922 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.406203 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.406268 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.406296 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.406336 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260926 05:49:23.406538 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.401759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35675 (local address 127.0.0.1:33064)
0926 05:49:23.402075 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.402087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.402149 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.402507 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.402511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.402514 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:23.402629 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.402632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.403697 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.403698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.404973 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.404977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.406039 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.406040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.406116 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.406119 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.406149 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.406242 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":8,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20260926 05:49:23.406390 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.401759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44477 (local address 127.0.0.1:33808)
0926 05:49:23.401947 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.401991 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.402075 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.402811 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.402812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.402814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:23.402855 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.402856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.404152 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.404154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.405228 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.405230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.405984 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.405985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.406075 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.406091 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.406192 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.406242 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260926 05:49:23.407063 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.402006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.11.247.2:35675)
0926 05:49:23.402268 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.402271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.402280 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:23.402317 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.402318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.402331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.402716 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:23.402805 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.403586 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.405090 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.405922 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.406792 (+ 870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.406866 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.406888 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.406956 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260926 05:49:23.407573 26124 tablet_service.cc:1467] Processing CreateTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 (DEFAULT_TABLE table=test-table [id=849deef5f1384440996c300b2666e2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:23.407599 26392 tablet_service.cc:1467] Processing CreateTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 (DEFAULT_TABLE table=test-table [id=849deef5f1384440996c300b2666e2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:23.408005 26124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:23.408005 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:23.411337 26258 tablet_service.cc:1467] Processing CreateTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 (DEFAULT_TABLE table=test-table [id=849deef5f1384440996c300b2666e2df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:23.411722 26258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:23.411728 26476 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap starting.
I20260926 05:49:23.412425 26476 tablet_bootstrap.cc:654] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:23.413100 26476 log.cc:824] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:23.413488 26475 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Bootstrap starting.
I20260926 05:49:23.413944 26476 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: No bootstrap required, opened a new log
I20260926 05:49:23.414028 26476 ts_tablet_manager.cc:1397] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:49:23.414211 26475 tablet_bootstrap.cc:654] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:23.414824 26475 log.cc:824] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:23.415611 26479 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Bootstrap starting.
I20260926 05:49:23.415681 26475 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: No bootstrap required, opened a new log
I20260926 05:49:23.415762 26475 ts_tablet_manager.cc:1397] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:49:23.416107 26476 raft_consensus.cc:348] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.416302 26476 raft_consensus.cc:374] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:23.416347 26476 raft_consensus.cc:729] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b074e08e0445b7be845fd2a1159f74, State: Initialized, Role: FOLLOWER
I20260926 05:49:23.416437 26479 tablet_bootstrap.cc:654] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:23.416513 26476 consensus_queue.cc:260] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.416857 26190 heartbeater.cc:503] Master 127.11.247.62:42097 was elected leader, sending a full tablet report...
I20260926 05:49:23.416868 26476 ts_tablet_manager.cc:1428] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:49:23.417003 26479 log.cc:824] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:23.417788 26479 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: No bootstrap required, opened a new log
I20260926 05:49:23.417862 26479 ts_tablet_manager.cc:1397] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:49:23.417755 26475 raft_consensus.cc:348] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.417935 26475 raft_consensus.cc:374] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:23.417976 26475 raft_consensus.cc:729] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8448dbaca7a84251928d864c13a86bff, State: Initialized, Role: FOLLOWER
I20260926 05:49:23.418131 26475 consensus_queue.cc:260] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.418416 26475 ts_tablet_manager.cc:1428] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:49:23.418550 26458 heartbeater.cc:503] Master 127.11.247.62:42097 was elected leader, sending a full tablet report...
I20260926 05:49:23.419917 26479 raft_consensus.cc:348] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.420066 26479 raft_consensus.cc:374] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:23.420113 26479 raft_consensus.cc:729] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa16fa2830964094abe6ae33a15015f5, State: Initialized, Role: FOLLOWER
I20260926 05:49:23.420228 26479 consensus_queue.cc:260] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.420544 26324 heartbeater.cc:503] Master 127.11.247.62:42097 was elected leader, sending a full tablet report...
I20260926 05:49:23.420550 26479 ts_tablet_manager.cc:1428] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 05:49:23.457723 26325 tablet.cc:2369] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:23.512056 26480 raft_consensus.cc:482] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:23.512198 26480 raft_consensus.cc:504] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.512634 26480 leader_election.cc:288] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
I20260926 05:49:23.517325 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.512944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:57423 (local address 127.11.247.2:35675)
0926 05:49:23.512994 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.512996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.513007 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:23.513107 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.513108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.513125 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.513198 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:23.513539 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.514411 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.516244 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.516863 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.516951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.516961 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.517199 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.517249 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:49:23.517325 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.513020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:48585 (local address 127.11.247.3:44477)
0926 05:49:23.513070 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.513073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.513156 (+ 83us) server_negotiation.cc:407] Connection header received
0926 05:49:23.513247 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.513248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.513260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.513304 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:23.513540 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.514410 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.516244 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.516863 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.516951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.516961 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.517199 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.517249 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:49:23.517354 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.512899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44477 (local address 127.11.247.1:48585)
0926 05:49:23.513072 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.513083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.513161 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.513328 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.513328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.513330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:23.513434 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.513435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.514550 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.514552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.516098 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.516103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.516969 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.516970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.517073 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.517080 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.517147 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.517206 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":99,"thread_start_us":125,"threads_started":1}
I20260926 05:49:23.517372 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.512886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35675 (local address 127.11.247.1:57423)
0926 05:49:23.512950 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.512963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.513011 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.513226 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.513227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.513229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:23.513394 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.513397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.514557 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.514557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.516098 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.516103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.516969 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.516970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.517073 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.517080 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.517147 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.517221 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":90}
I20260926 05:49:23.518072 26278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ec5ef3c9c4d86ab760dd0a5ee4f07" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa16fa2830964094abe6ae33a15015f5" is_pre_election: true
I20260926 05:49:23.518065 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ec5ef3c9c4d86ab760dd0a5ee4f07" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8448dbaca7a84251928d864c13a86bff" is_pre_election: true
I20260926 05:49:23.518250 26278 raft_consensus.cc:2454] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 0.
I20260926 05:49:23.518254 26412 raft_consensus.cc:2454] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 0.
I20260926 05:49:23.518533 26078 leader_election.cc:302] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74, 8448dbaca7a84251928d864c13a86bff; no voters:
I20260926 05:49:23.518733 26480 raft_consensus.cc:2781] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:49:23.518843 26480 raft_consensus.cc:482] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:49:23.518885 26480 raft_consensus.cc:3037] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:23.519701 26480 raft_consensus.cc:504] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.519891 26480 leader_election.cc:288] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 election: Requested vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
I20260926 05:49:23.520078 26278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ec5ef3c9c4d86ab760dd0a5ee4f07" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa16fa2830964094abe6ae33a15015f5"
I20260926 05:49:23.520200 26278 raft_consensus.cc:3037] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:23.520153 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ec5ef3c9c4d86ab760dd0a5ee4f07" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8448dbaca7a84251928d864c13a86bff"
I20260926 05:49:23.520396 26412 raft_consensus.cc:3037] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:23.521225 26278 raft_consensus.cc:2454] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 1.
I20260926 05:49:23.521338 26412 raft_consensus.cc:2454] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 1.
I20260926 05:49:23.521423 26079 leader_election.cc:302] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74, fa16fa2830964094abe6ae33a15015f5; no voters:
I20260926 05:49:23.521541 26480 raft_consensus.cc:2781] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:23.521674 26480 raft_consensus.cc:686] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 LEADER]: Becoming Leader. State: Replica: 30b074e08e0445b7be845fd2a1159f74, State: Running, Role: LEADER
I20260926 05:49:23.521781 26480 consensus_queue.cc:237] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.522773 25998 catalog_manager.cc:5495] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "30b074e08e0445b7be845fd2a1159f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } health_report { overall_health: HEALTHY } } }
W20260926 05:49:23.538745 26191 tablet.cc:2369] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:23.556874 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.553975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.11.247.1:40061)
0926 05:49:23.554055 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.554058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.554067 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:23.554119 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.554120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.554145 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.554245 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:49:23.554419 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.554998 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.555933 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.556172 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.556258 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.556447 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:49:23.556573 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:49:23.556683 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.556723 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.556746 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 05:49:23.558923 26412 raft_consensus.cc:1261] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Refusing update from remote peer 30b074e08e0445b7be845fd2a1159f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:23.558965 26278 raft_consensus.cc:1261] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Refusing update from remote peer 30b074e08e0445b7be845fd2a1159f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:23.559283 26480 consensus_queue.cc:1035] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:23.559486 26485 consensus_queue.cc:1035] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:23.562145 26490 mvcc.cc:205] Tried to move back new op lower bound from 7333485623534510080 to 7333485623386148864. Current Snapshot: MvccSnapshot[applied={T|T < 7333485623534510080}]
I20260926 05:49:23.563103 26489 mvcc.cc:205] Tried to move back new op lower bound from 7333485623534510080 to 7333485623386148864. Current Snapshot: MvccSnapshot[applied={T|T < 7333485623534510080 or (T in {7333485623534510080})}]
I20260926 05:49:23.564248 25998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:23.564357 25998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:23.565241 25998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Sending DeleteTablet for 3 replicas of tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07
I20260926 05:49:23.565658 26258 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:23 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:33064
I20260926 05:49:23.565666 26392 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:23 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:33808
I20260926 05:49:23.565922 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26061
I20260926 05:49:23.565963 26498 tablet_replica.cc:330] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: stopping tablet replica
I20260926 05:49:23.565953 26124 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:23 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:57526
I20260926 05:49:23.566066 26498 raft_consensus.cc:2229] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:23.566165 26498 raft_consensus.cc:2258] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:23.567371 26499 tablet_replica.cc:330] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: stopping tablet replica
I20260926 05:49:23.567451 26499 raft_consensus.cc:2229] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:23.567524 26499 raft_consensus.cc:2258] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:23.567996 26499 ts_tablet_manager.cc:1905] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:23.568167 26498 ts_tablet_manager.cc:1905] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:23.569703 26498 ts_tablet_manager.cc:1918] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:23.569765 26498 log.cc:1196] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/eb7ec5ef3c9c4d86ab760dd0a5ee4f07
I20260926 05:49:23.569999 26498 ts_tablet_manager.cc:1939] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P fa16fa2830964094abe6ae33a15015f5: Deleting consensus metadata
I20260926 05:49:23.570021 26499 ts_tablet_manager.cc:1918] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:23.570088 26499 log.cc:1196] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/eb7ec5ef3c9c4d86ab760dd0a5ee4f07
I20260926 05:49:23.570345 26499 ts_tablet_manager.cc:1939] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 8448dbaca7a84251928d864c13a86bff: Deleting consensus metadata
I20260926 05:49:23.570395 25985 catalog_manager.cc:4841] TS fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 (table test-table [id=849deef5f1384440996c300b2666e2df]) successfully deleted
I20260926 05:49:23.570930 25984 catalog_manager.cc:4841] TS 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 (table test-table [id=849deef5f1384440996c300b2666e2df]) successfully deleted
W20260926 05:49:23.572417 25984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:40061 (error 108)
I20260926 05:49:23.572580 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26195
I20260926 05:49:23.572975 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.572858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:40061 (local address 127.0.0.1:57550)
0926 05:49:23.572908 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.572936 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:49:23.573113 25984 catalog_manager.cc:4584] TS 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1:40061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07: Network error: Client connection negotiation failed: client connection to 127.11.247.1:40061: connect: Connection refused (error 111)
I20260926 05:49:23.578392 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26329
I20260926 05:49:23.583498 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 25967
I20260926 05:49:23.589741 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42097
--webserver_interface=127.11.247.62
--webserver_port=34341
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:23.672549 26501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:23.672813 26501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:23.672856 26501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:23.674027 26501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:23.674109 26501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:23.674134 26501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:23.674163 26501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:23.674196 26501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:23.675767 26501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:23.676060 26501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:23.676301 26501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:23.686771 26510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.686771 26508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.686771 26507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.686810 26501 server_base.cc:1034] running on GCE node
I20260926 05:49:23.687244 26501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:23.687584 26501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:23.688776 26501 hybrid_clock.cc:648] HybridClock initialized: now 1790401763688756 us; error 34 us; skew 500 ppm
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:23.691495 26501 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:23.692019 26501 webserver.cc:460] Webserver started at http://127.11.247.62:34341/ using document root <none> and password file <none>
I20260926 05:49:23.692291 26501 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:23.692376 26501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:23.693815 26501 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:49:23.694658 26517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.694864 26501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:23.694967 26501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "d7de304b39c94acf8c695ea5848d8ca1"
format_stamp: "Formatted at 2026-09-26 05:49:22 on dist-test-slave-21p4"
server_key: "1269e7004c9cb86cf70e72bf972c4dcc"
server_key_iv: "1214bc94430068d6a5de8aca2aa49f12"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.695400 26501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:23.706087 26501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:23.708711 26501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:23.708932 26501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:23.713378 26501 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:42097
I20260926 05:49:23.713439 26569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:42097 every 8 connection(s)
I20260926 05:49:23.713888 26501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:23.714840 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26501
I20260926 05:49:23.715147 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:40061
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:42097
--webserver_port=38907
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:23.715775 26570 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:49:23.716454 26570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap starting.
I20260926 05:49:23.725126 26570 log.cc:824] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:23.727361 26570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:49:23.727705 26570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap complete.
I20260926 05:49:23.730101 26570 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:23.730329 26570 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7de304b39c94acf8c695ea5848d8ca1, State: Initialized, Role: FOLLOWER
I20260926 05:49:23.730504 26570 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:23.730616 26570 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:23.730677 26570 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:23.730736 26570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:49:23.731822 26570 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:23.731920 26570 leader_election.cc:302] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7de304b39c94acf8c695ea5848d8ca1; no voters:
I20260926 05:49:23.732142 26570 leader_election.cc:288] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:49:23.732187 26574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:49:23.732360 26574 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 LEADER]: Becoming Leader. State: Replica: d7de304b39c94acf8c695ea5848d8ca1, State: Running, Role: LEADER
I20260926 05:49:23.732426 26570 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:23.732518 26574 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:23.732812 26574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7de304b39c94acf8c695ea5848d8ca1. Latest consensus state: current_term: 2 leader_uuid: "d7de304b39c94acf8c695ea5848d8ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } } }
I20260926 05:49:23.732818 26576 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7de304b39c94acf8c695ea5848d8ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } } }
I20260926 05:49:23.733039 26576 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:23.733461 26574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:23.733485 26583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:23.734447 26583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:23.734556 26583 catalog_manager.cc:1259] Loaded cluster ID: aa50ff9651c747b38f5f85bcc9441333
I20260926 05:49:23.734611 26583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:49:23.735059 26583 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 05:49:23.737738 26583 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 05:49:23.739683 26583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:23.739887 26583 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Loaded TSK: 0
I20260926 05:49:23.740108 26583 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:49:23.806813 26572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:23.807066 26572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:23.807107 26572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:23.808354 26572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:23.808432 26572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:23.808480 26572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:49:23.810068 26572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:40061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=38907
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:23.810346 26572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:23.810611 26572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:23.815696 26597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.815696 26600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.815696 26598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.816375 26572 server_base.cc:1034] running on GCE node
I20260926 05:49:23.816673 26572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:23.816929 26572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:23.818094 26572 hybrid_clock.cc:648] HybridClock initialized: now 1790401763818081 us; error 32 us; skew 500 ppm
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:23.820798 26572 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:49:23.821317 26572 webserver.cc:460] Webserver started at http://127.11.247.1:38907/ using document root <none> and password file <none>
I20260926 05:49:23.821563 26572 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:23.821646 26572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:23.823243 26572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:49:23.824019 26607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.824191 26572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:23.824280 26572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "30b074e08e0445b7be845fd2a1159f74"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "b8322ba523f0debf330ea9f43f7f1ba9"
server_key_iv: "4b07b4b364f3d2c8116829e311069bc4"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.824658 26572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:23.839519 26572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:23.842005 26572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:23.842208 26572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:23.842485 26572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:23.843137 26614 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:49:23.844674 26572 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:49:23.844735 26572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:49:23.844765 26572 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:49:23.845477 26572 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:49:23.845525 26572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:49:23.845573 26614 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap starting.
I20260926 05:49:23.855657 26572 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:40061
I20260926 05:49:23.856236 26572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:49:23.858551 26614 log.cc:824] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:23.859220 26721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:40061 every 8 connection(s)
I20260926 05:49:23.861346 26614 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:49:23.861658 26614 tablet_bootstrap.cc:492] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap complete.
I20260926 05:49:23.861989 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26572
I20260926 05:49:23.862283 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:35675
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:42097
--webserver_port=41575
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:23.862983 26614 ts_tablet_manager.cc:1397] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.007s
I20260926 05:49:23.865027 26614 raft_consensus.cc:348] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.865396 26614 raft_consensus.cc:729] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b074e08e0445b7be845fd2a1159f74, State: Initialized, Role: FOLLOWER
I20260926 05:49:23.865562 26614 consensus_queue.cc:260] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:23.865906 26614 ts_tablet_manager.cc:1428] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:23.875679 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.859483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:48385)
0926 05:49:23.862742 (+ 3259us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.862801 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.863629 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.864319 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.864324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.864726 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:23.865030 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.865038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.866695 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.866696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.867873 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.867876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.868696 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.868697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.868843 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.869515 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:23.869524 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:23.871371 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:23.873608 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.873617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.873620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.873882 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.874201 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:23.874202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:23.874203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:23.874330 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:23.874900 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:23.874907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:23.875078 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.875208 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.875429 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3204,"thread_start_us":107,"threads_started":1}
I20260926 05:49:23.875694 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.859841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:48385 (local address 127.11.247.62:42097)
0926 05:49:23.860048 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.860118 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.862956 (+ 2838us) server_negotiation.cc:407] Connection header received
0926 05:49:23.863893 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.863903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.863973 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.864093 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:23.865179 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.866574 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.868015 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.868568 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.868721 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.871533 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:23.871537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:23.871537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:23.871549 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:23.873483 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.874007 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.874010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.874011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.874121 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:23.874428 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:23.874429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:23.874430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:23.874792 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:23.874922 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.875268 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.875438 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260926 05:49:23.876961 26722 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42097
I20260926 05:49:23.877095 26722 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:23.877295 26722 heartbeater.cc:511] Master 127.11.247.62:42097 requested a full tablet report, sending...
I20260926 05:49:23.878106 26534 ts_manager.cc:194] Registered new tserver with Master: 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1:40061)
I20260926 05:49:23.879343 26534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:48385
I20260926 05:49:23.880548 26722 heartbeater.cc:503] Master 127.11.247.62:42097 was elected leader, sending a full tablet report...
I20260926 05:49:23.884174 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.879699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.11.247.1:40061)
0926 05:49:23.880168 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.880173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.880182 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:23.880217 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.880219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.880364 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.880446 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:23.880688 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.881847 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.883173 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.883695 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.883789 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.883864 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:23.884044 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 05:49:23.884088 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":100,"threads_started":1}
I20260926 05:49:23.884161 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.878763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:40061 (local address 127.0.0.1:57554)
0926 05:49:23.879914 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.879954 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.880023 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.880525 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.880528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.880530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:23.880598 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.880600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.881943 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.881944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.883063 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:23.883071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:23.883804 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:23.883805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:23.883902 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:23.883916 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:49:23.884013 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:49:23.884077 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":100,"threads_started":1}
I20260926 05:49:23.884958 26656 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:23 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260926 05:49:23.885623 26731 tablet_replica.cc:330] stopping tablet replica
I20260926 05:49:23.885756 26731 raft_consensus.cc:2229] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:23.885834 26731 raft_consensus.cc:2258] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:23.886191 26731 ts_tablet_manager.cc:1905] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:23.888926 26731 ts_tablet_manager.cc:1918] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:23.889027 26731 log.cc:1196] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/eb7ec5ef3c9c4d86ab760dd0a5ee4f07
I20260926 05:49:23.889319 26731 ts_tablet_manager.cc:1939] T eb7ec5ef3c9c4d86ab760dd0a5ee4f07 P 30b074e08e0445b7be845fd2a1159f74: Deleting consensus metadata
I20260926 05:49:23.889983 26519 catalog_manager.cc:4841] TS 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1:40061): tablet eb7ec5ef3c9c4d86ab760dd0a5ee4f07 (table test-table [id=849deef5f1384440996c300b2666e2df]) successfully deleted
W20260926 05:49:23.954967 26727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:23.955194 26727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:23.955220 26727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:23.956359 26727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:23.956396 26727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:23.956424 26727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:49:23.957890 26727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:35675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=41575
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:23.958104 26727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:23.958280 26727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:23.963505 26739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.963510 26737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:23.963541 26736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:23.963929 26727 server_base.cc:1034] running on GCE node
I20260926 05:49:23.964092 26727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:23.964298 26727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:23.965437 26727 hybrid_clock.cc:648] HybridClock initialized: now 1790401763965425 us; error 26 us; skew 500 ppm
Sep 26 05:49:23 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:23.967962 26727 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:49:23.968503 26727 webserver.cc:460] Webserver started at http://127.11.247.2:41575/ using document root <none> and password file <none>
I20260926 05:49:23.968760 26727 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:23.968845 26727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:23.970270 26727 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:49:23.970960 26746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.971144 26727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:49:23.971243 26727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "fa16fa2830964094abe6ae33a15015f5"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "2ec7871d6dae41b58ec86f3eed5938e4"
server_key_iv: "de818e98a849c55cb4331275c948de82"
server_key_version: "encryptionkey@0"
I20260926 05:49:23.971613 26727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:23.987701 26727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:23.989885 26727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:23.990061 26727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:23.990310 26727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:23.990674 26727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:23.990732 26727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.990777 26727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:23.990814 26727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:23.997068 26727 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35675
I20260926 05:49:23.997112 26859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35675 every 8 connection(s)
I20260926 05:49:23.997539 26727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:49:24.001024 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26727
I20260926 05:49:24.001302 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:44477
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:42097
--webserver_port=41325
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:24.008606 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.998269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:38787 (local address 127.11.247.62:42097)
0926 05:49:23.998325 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:23.998327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:23.998646 (+ 319us) server_negotiation.cc:407] Connection header received
0926 05:49:23.999196 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:23.999196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:23.999210 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:23.999272 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:24.000250 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.000752 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.001935 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.002286 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.002366 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.004689 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:24.004691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:24.004691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:24.004700 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:24.006895 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:24.007365 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:24.007367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:24.007367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:24.007438 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:24.007741 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:24.007742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:24.007743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:24.007916 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:24.007995 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.008323 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.008451 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:49:24.008639 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:23.998255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:38787)
0926 05:49:23.998486 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:23.998528 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:23.999107 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:23.999339 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:23.999342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:23.999966 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:24.000161 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.000166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.000857 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.000858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.001846 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.001849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.002415 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.002416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.002528 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.003191 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:24.003200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:24.004588 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:24.007004 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:24.007011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:24.007013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:24.007274 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:24.007550 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:24.007551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:24.007551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:24.007668 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:24.008009 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:24.008016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:24.008171 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 05:49:24.008272 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 05:49:24.008456 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260926 05:49:24.009239 26860 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42097
I20260926 05:49:24.009370 26860 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:24.009595 26860 heartbeater.cc:511] Master 127.11.247.62:42097 requested a full tablet report, sending...
I20260926 05:49:24.010038 26534 ts_manager.cc:194] Registered new tserver with Master: fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675)
I20260926 05:49:24.010421 26534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:38787
W20260926 05:49:24.085682 26863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:24.085891 26863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:24.085917 26863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:24.087306 26863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:24.087353 26863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:24.087381 26863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:49:24.088997 26863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:44477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=41325
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:24.089258 26863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:24.089524 26863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:24.094993 26871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.094969 26868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.094981 26869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.095201 26863 server_base.cc:1034] running on GCE node
I20260926 05:49:24.095413 26863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:24.095649 26863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:24.096794 26863 hybrid_clock.cc:648] HybridClock initialized: now 1790401764096780 us; error 30 us; skew 500 ppm
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:24.099598 26863 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:49:24.100104 26863 webserver.cc:460] Webserver started at http://127.11.247.3:41325/ using document root <none> and password file <none>
I20260926 05:49:24.100339 26863 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:24.100421 26863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:24.101872 26863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:24.102540 26878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.102746 26863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:24.102855 26863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "8448dbaca7a84251928d864c13a86bff"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "c24476daa21f8f176ff08c4eec443306"
server_key_iv: "a2cf18b8c55c32b937998e2693707add"
server_key_version: "encryptionkey@0"
I20260926 05:49:24.103266 26863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:24.115129 26863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:24.117594 26863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:24.117708 26863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:24.117916 26863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:24.118173 26863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:24.118203 26863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.118240 26863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:24.118252 26863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.124117 26863 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:44477
I20260926 05:49:24.124182 26991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:44477 every 8 connection(s)
I20260926 05:49:24.124516 26863 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:49:24.126832 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 26863
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:24.134696 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.127244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.11.247.62:42097)
0926 05:49:24.127340 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.127342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.127368 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:49:24.127434 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.127435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.127445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.127506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:24.128189 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.128647 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.129654 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.129888 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.129935 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.131502 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:24.131504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:24.131505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:24.131513 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:24.133443 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:24.133818 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:24.133819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:24.133820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:24.133880 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:24.134089 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:24.134091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:24.134091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:24.134245 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:24.134407 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.134481 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.134544 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":53,"threads_started":1}
I20260926 05:49:24.135809 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.125309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:32997 (local address 127.11.247.62:42097)
0926 05:49:24.125455 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.125457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.125653 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:49:24.126150 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.126152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.126163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.126225 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:24.127395 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.128123 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.129438 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.129657 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.129725 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.132575 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:24.132576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:24.132576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:24.132583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:24.134189 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:24.134689 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:24.134690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:24.134690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:24.134775 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:24.135102 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:24.135103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:24.135104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:24.135258 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:24.135333 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.135678 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.135752 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 05:49:24.136003 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.125310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:32997)
0926 05:49:24.125482 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.125527 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.126051 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.126281 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:24.126284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:24.127020 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:24.127300 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.127306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.128238 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.128243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.129351 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.129354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.129746 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.129747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.129854 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.130417 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:24.130427 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:24.132486 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:24.134264 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:24.134270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:24.134272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:24.134627 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:24.134871 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:24.134872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:24.134872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:24.135017 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:24.135326 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:24.135332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:24.135525 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 05:49:24.135627 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:49:24.135757 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260926 05:49:24.136472 26992 heartbeater.cc:347] Connected to a master server at 127.11.247.62:42097
I20260926 05:49:24.136595 26992 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:24.136744 26992 heartbeater.cc:511] Master 127.11.247.62:42097 requested a full tablet report, sending...
I20260926 05:49:24.137120 26534 ts_manager.cc:194] Registered new tserver with Master: 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
I20260926 05:49:24.137503 26534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:32997
I20260926 05:49:24.146517 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:49:24.149430 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.146966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.11.247.62:42097)
0926 05:49:24.147040 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.147044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.147105 (+ 61us) server_negotiation.cc:407] Connection header received
0926 05:49:24.147179 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.147179 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.147214 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.147274 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:49:24.147396 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.147883 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.148682 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.148882 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.148947 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.149084 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:49:24.149191 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:49:24.149268 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.149283 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.149301 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 05:49:24.149849 26534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:49:24.151618 26534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:49:24.154124 26656 tablet_service.cc:1467] Processing CreateTablet for tablet 870338f3f6554976ae69d53f4a9eb529 (DEFAULT_TABLE table=test-table [id=fa3caaef8d2348ca8ed140ba9037589c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:24.154345 26656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870338f3f6554976ae69d53f4a9eb529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:24.156018 26614 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap starting.
I20260926 05:49:24.156407 26614 tablet_bootstrap.cc:654] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:24.157106 26614 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: No bootstrap required, opened a new log
I20260926 05:49:24.157159 26614 ts_tablet_manager.cc:1397] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:49:24.157303 26614 raft_consensus.cc:348] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.157384 26614 raft_consensus.cc:374] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:24.157435 26614 raft_consensus.cc:729] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b074e08e0445b7be845fd2a1159f74, State: Initialized, Role: FOLLOWER
I20260926 05:49:24.157481 26614 consensus_queue.cc:260] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.157598 26614 ts_tablet_manager.cc:1428] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.157866 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.153628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35675 (local address 127.0.0.1:33066)
0926 05:49:24.153679 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.153690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.153730 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.154065 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:24.154066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:24.154067 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:24.154116 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.154118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.155388 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.155389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.156582 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.156585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.157610 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.157611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.157685 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.157689 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:49:24.157724 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 05:49:24.157770 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":25}
I20260926 05:49:24.157980 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.153656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44477 (local address 127.0.0.1:33818)
0926 05:49:24.153914 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.153925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.153970 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.154289 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:24.154289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:24.154292 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:24.154339 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.154341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.155632 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.155632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.156690 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.156692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.157679 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.157680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.157753 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.157756 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:49:24.157833 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:49:24.157876 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260926 05:49:24.158192 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.153754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.11.247.3:44477)
0926 05:49:24.154043 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.154048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.154059 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:24.154098 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.154100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.154117 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.154196 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:24.154452 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.155517 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.156781 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.157508 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.157579 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.157643 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.157850 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.157891 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260926 05:49:24.158272 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.153723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.11.247.2:35675)
0926 05:49:24.153892 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.153897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.153908 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:24.153953 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.153955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.153984 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.154070 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:24.154172 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.155284 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.156673 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.157489 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.158035 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.158104 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.158131 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.158182 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260926 05:49:24.159049 26794 tablet_service.cc:1467] Processing CreateTablet for tablet 870338f3f6554976ae69d53f4a9eb529 (DEFAULT_TABLE table=test-table [id=fa3caaef8d2348ca8ed140ba9037589c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:24.159035 26926 tablet_service.cc:1467] Processing CreateTablet for tablet 870338f3f6554976ae69d53f4a9eb529 (DEFAULT_TABLE table=test-table [id=fa3caaef8d2348ca8ed140ba9037589c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:24.159379 26926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870338f3f6554976ae69d53f4a9eb529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:24.159379 26794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870338f3f6554976ae69d53f4a9eb529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:24.162930 27002 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Bootstrap starting.
I20260926 05:49:24.163524 27002 tablet_bootstrap.cc:654] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:24.163996 27002 log.cc:824] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:24.164324 27003 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Bootstrap starting.
I20260926 05:49:24.164718 27002 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: No bootstrap required, opened a new log
I20260926 05:49:24.164773 27002 ts_tablet_manager.cc:1397] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:49:24.164907 27003 tablet_bootstrap.cc:654] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:24.165405 27003 log.cc:824] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:24.166100 27003 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: No bootstrap required, opened a new log
I20260926 05:49:24.166181 27003 ts_tablet_manager.cc:1397] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:49:24.166476 27002 raft_consensus.cc:348] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.166671 27002 raft_consensus.cc:374] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:24.166699 27002 raft_consensus.cc:729] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa16fa2830964094abe6ae33a15015f5, State: Initialized, Role: FOLLOWER
I20260926 05:49:24.166833 27002 consensus_queue.cc:260] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.167124 27002 ts_tablet_manager.cc:1428] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:49:24.167146 26860 heartbeater.cc:503] Master 127.11.247.62:42097 was elected leader, sending a full tablet report...
I20260926 05:49:24.168136 27003 raft_consensus.cc:348] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.168331 27003 raft_consensus.cc:374] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:24.168378 27003 raft_consensus.cc:729] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8448dbaca7a84251928d864c13a86bff, State: Initialized, Role: FOLLOWER
I20260926 05:49:24.168489 27003 consensus_queue.cc:260] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.168751 27003 ts_tablet_manager.cc:1428] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:49:24.168794 26992 heartbeater.cc:503] Master 127.11.247.62:42097 was elected leader, sending a full tablet report...
W20260926 05:49:24.248149 26861 tablet.cc:2369] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:24.262826 26728 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:24.263049 26728 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.263473 26728 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
I20260926 05:49:24.266458 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.263849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:40535 (local address 127.11.247.3:44477)
0926 05:49:24.263919 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.263921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.263930 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:49:24.263973 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.263974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.264001 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.264086 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:24.264250 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.264741 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.265697 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.266088 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.266173 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.266183 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.266343 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.266384 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:49:24.266476 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.263703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44477 (local address 127.11.247.1:40535)
0926 05:49:24.263777 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.263792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.263868 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.264097 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:24.264098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:24.264100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:24.264177 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.264179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.264833 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.264834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.265600 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.265602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.266184 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.266185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.266268 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.266271 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:49:24.266336 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:49:24.266384 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 05:49:24.266829 26946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870338f3f6554976ae69d53f4a9eb529" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8448dbaca7a84251928d864c13a86bff" is_pre_election: true
I20260926 05:49:24.266930 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.263712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35675 (local address 127.11.247.1:60629)
0926 05:49:24.263981 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.263989 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.264080 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.264251 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:24.264252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:24.264253 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:24.264588 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.264590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.265233 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.265234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.266011 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:24.266013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.266646 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.266647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:24.266717 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.266722 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:49:24.266783 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:49:24.266818 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":82,"thread_start_us":89,"threads_started":1}
I20260926 05:49:24.266912 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.263829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:60629 (local address 127.11.247.2:35675)
0926 05:49:24.263924 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.263927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.264063 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:49:24.264152 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.264153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.264162 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.264233 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:24.264672 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.265145 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.266129 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.266508 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.266635 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.266645 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.266801 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.266841 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 05:49:24.267009 26946 raft_consensus.cc:2454] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 0.
I20260926 05:49:24.267270 26609 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74, 8448dbaca7a84251928d864c13a86bff; no voters:
I20260926 05:49:24.267298 26814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870338f3f6554976ae69d53f4a9eb529" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa16fa2830964094abe6ae33a15015f5" is_pre_election: true
I20260926 05:49:24.267444 26728 raft_consensus.cc:2781] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:49:24.267452 26814 raft_consensus.cc:2454] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 0.
I20260926 05:49:24.267504 26728 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:49:24.267549 26728 raft_consensus.cc:3037] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:24.268337 26728 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.268515 26728 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 election: Requested vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
I20260926 05:49:24.268692 26814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870338f3f6554976ae69d53f4a9eb529" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa16fa2830964094abe6ae33a15015f5"
I20260926 05:49:24.268760 26946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870338f3f6554976ae69d53f4a9eb529" candidate_uuid: "30b074e08e0445b7be845fd2a1159f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8448dbaca7a84251928d864c13a86bff"
I20260926 05:49:24.268793 26814 raft_consensus.cc:3037] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:24.268874 26946 raft_consensus.cc:3037] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:24.269605 26814 raft_consensus.cc:2454] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 1.
I20260926 05:49:24.269810 26610 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74, fa16fa2830964094abe6ae33a15015f5; no voters:
I20260926 05:49:24.269829 26946 raft_consensus.cc:2454] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b074e08e0445b7be845fd2a1159f74 in term 1.
I20260926 05:49:24.269884 26728 raft_consensus.cc:2781] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:24.270059 26728 raft_consensus.cc:686] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 LEADER]: Becoming Leader. State: Replica: 30b074e08e0445b7be845fd2a1159f74, State: Running, Role: LEADER
I20260926 05:49:24.270143 26728 consensus_queue.cc:237] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.270972 26534 catalog_manager.cc:5495] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "30b074e08e0445b7be845fd2a1159f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } health_report { overall_health: HEALTHY } } }
I20260926 05:49:24.303391 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.301106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.11.247.1:40061)
0926 05:49:24.301179 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:24.301181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:24.301193 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:49:24.301238 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:24.301239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:24.301257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:24.301334 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:49:24.301470 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.301941 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.302649 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:24.302839 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:24.302891 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:24.303016 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:49:24.303110 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:49:24.303161 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:24.303246 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 05:49:24.303294 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:49:24.305238 26946 raft_consensus.cc:1261] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Refusing update from remote peer 30b074e08e0445b7be845fd2a1159f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:24.305310 26814 raft_consensus.cc:1261] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Refusing update from remote peer 30b074e08e0445b7be845fd2a1159f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:24.305634 26728 consensus_queue.cc:1035] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:24.305785 27009 consensus_queue.cc:1035] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:24.308508 27011 mvcc.cc:205] Tried to move back new op lower bound from 7333485626590978048 to 7333485626451165184. Current Snapshot: MvccSnapshot[applied={T|T < 7333485626590978048}]
I20260926 05:49:24.309546 27013 mvcc.cc:205] Tried to move back new op lower bound from 7333485626590978048 to 7333485626451165184. Current Snapshot: MvccSnapshot[applied={T|T < 7333485626590978048 or (T in {7333485626590978048})}]
I20260926 05:49:24.311036 26534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:24.311137 26534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:24.311939 26534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Sending DeleteTablet for 3 replicas of tablet 870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:24.312330 26926 tablet_service.cc:1514] Processing DeleteTablet for tablet 870338f3f6554976ae69d53f4a9eb529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:24 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:33818
I20260926 05:49:24.312314 26656 tablet_service.cc:1514] Processing DeleteTablet for tablet 870338f3f6554976ae69d53f4a9eb529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:24 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260926 05:49:24.312356 26794 tablet_service.cc:1514] Processing DeleteTablet for tablet 870338f3f6554976ae69d53f4a9eb529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:24 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:33066
I20260926 05:49:24.312474 26731 tablet_replica.cc:330] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: stopping tablet replica
I20260926 05:49:24.312529 26731 raft_consensus.cc:2229] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:49:24.312551 27021 tablet_replica.cc:330] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: stopping tablet replica
I20260926 05:49:24.312623 26731 raft_consensus.cc:2258] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:24.312636 27021 raft_consensus.cc:2229] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:24.312734 27021 raft_consensus.cc:2258] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:24.312830 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26572
I20260926 05:49:24.313586 27021 ts_tablet_manager.cc:1905] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:24.314601 27022 tablet_replica.cc:330] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: stopping tablet replica
I20260926 05:49:24.314771 27022 raft_consensus.cc:2229] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:24.314926 27022 raft_consensus.cc:2258] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:24.315289 27021 ts_tablet_manager.cc:1918] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:24.315375 27021 log.cc:1196] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:24.315490 27022 ts_tablet_manager.cc:1905] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:24.315703 27021 ts_tablet_manager.cc:1939] T 870338f3f6554976ae69d53f4a9eb529 P 8448dbaca7a84251928d864c13a86bff: Deleting consensus metadata
I20260926 05:49:24.316126 26519 catalog_manager.cc:4841] TS 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): tablet 870338f3f6554976ae69d53f4a9eb529 (table test-table [id=fa3caaef8d2348ca8ed140ba9037589c]) successfully deleted
I20260926 05:49:24.317664 27022 ts_tablet_manager.cc:1918] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:24.317752 27022 log.cc:1196] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:24.318060 27022 ts_tablet_manager.cc:1939] T 870338f3f6554976ae69d53f4a9eb529 P fa16fa2830964094abe6ae33a15015f5: Deleting consensus metadata
I20260926 05:49:24.318553 26520 catalog_manager.cc:4841] TS fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): tablet 870338f3f6554976ae69d53f4a9eb529 (table test-table [id=fa3caaef8d2348ca8ed140ba9037589c]) successfully deleted
W20260926 05:49:24.320019 26519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:40061 (error 108)
I20260926 05:49:24.320214 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26727
I20260926 05:49:24.320523 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.320400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:40061 (local address 127.0.0.1:57576)
0926 05:49:24.320454 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.320486 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:49:24.320919 26519 catalog_manager.cc:4584] TS 30b074e08e0445b7be845fd2a1159f74 (127.11.247.1:40061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 870338f3f6554976ae69d53f4a9eb529: Network error: Client connection negotiation failed: client connection to 127.11.247.1:40061: connect: Connection refused (error 111)
I20260926 05:49:24.328613 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26863
I20260926 05:49:24.333201 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 26501
I20260926 05:49:24.338881 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42097
--webserver_interface=127.11.247.62
--webserver_port=34341
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:24.420791 27024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:24.421000 27024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:24.421023 27024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:24.422192 27024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:24.422235 27024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:24.422250 27024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:24.422261 27024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:24.422297 27024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:24.423964 27024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:24.424224 27024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:24.424449 27024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:24.429472 27029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.429608 27024 server_base.cc:1034] running on GCE node
W20260926 05:49:24.429673 27030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.429666 27032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.430133 27024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:24.430437 27024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:24.431597 27024 hybrid_clock.cc:648] HybridClock initialized: now 1790401764431598 us; error 36 us; skew 500 ppm
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:24.433965 27024 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:24.434444 27024 webserver.cc:460] Webserver started at http://127.11.247.62:34341/ using document root <none> and password file <none>
I20260926 05:49:24.434721 27024 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:24.434792 27024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:24.436170 27024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:24.436837 27039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.437001 27024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:49:24.437116 27024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "d7de304b39c94acf8c695ea5848d8ca1"
format_stamp: "Formatted at 2026-09-26 05:49:22 on dist-test-slave-21p4"
server_key: "1269e7004c9cb86cf70e72bf972c4dcc"
server_key_iv: "1214bc94430068d6a5de8aca2aa49f12"
server_key_version: "encryptionkey@0"
I20260926 05:49:24.437474 27024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:24.460752 27024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:24.462941 27024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:24.463127 27024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:24.467149 27024 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:42097
I20260926 05:49:24.467202 27091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:42097 every 8 connection(s)
I20260926 05:49:24.467625 27024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:24.469328 27092 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:49:24.469931 27092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap starting.
I20260926 05:49:24.473914 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27024
I20260926 05:49:24.474216 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:40061
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:42097
--webserver_port=38907
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:24.478673 27092 log.cc:824] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:24.481016 27092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:49:24.481370 27092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1: Bootstrap complete.
I20260926 05:49:24.483814 27092 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:24.484072 27092 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7de304b39c94acf8c695ea5848d8ca1, State: Initialized, Role: FOLLOWER
I20260926 05:49:24.484256 27092 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:24.484337 27092 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:24.484397 27092 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:24.484445 27092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:49:24.485457 27092 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:24.485584 27092 leader_election.cc:302] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7de304b39c94acf8c695ea5848d8ca1; no voters:
I20260926 05:49:24.485818 27092 leader_election.cc:288] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 05:49:24.485882 27096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:49:24.486034 27096 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [term 3 LEADER]: Becoming Leader. State: Replica: d7de304b39c94acf8c695ea5848d8ca1, State: Running, Role: LEADER
I20260926 05:49:24.486099 27092 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:24.486142 27096 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } }
I20260926 05:49:24.486408 27096 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7de304b39c94acf8c695ea5848d8ca1. Latest consensus state: current_term: 3 leader_uuid: "d7de304b39c94acf8c695ea5848d8ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } } }
I20260926 05:49:24.486476 27096 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:24.486618 27097 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d7de304b39c94acf8c695ea5848d8ca1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de304b39c94acf8c695ea5848d8ca1" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42097 } } }
I20260926 05:49:24.486676 27097 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7de304b39c94acf8c695ea5848d8ca1 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:24.486785 27101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:24.487634 27101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:24.487740 27101 catalog_manager.cc:1259] Loaded cluster ID: aa50ff9651c747b38f5f85bcc9441333
I20260926 05:49:24.487783 27101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 05:49:24.488236 27101 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb6a68c29ad google::LogMessage::Fail() at ??:0
@ 0x7fb6a68c588a google::LogMessage::SendToLog() at ??:0
@ 0x7fb6a68c24de google::LogMessage::Flush() at ??:0
@ 0x7fb6a68c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb6ae7e30f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb6ae7ec8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb6a87900d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb6a8785954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb6a95e96db start_thread at ??:0
@ 0x7fb6a53c061f clone at ??:0
W20260926 05:49:24.573557 27094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:24.573832 27094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:24.573875 27094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:24.575163 27094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:24.575232 27094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:24.575282 27094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:49:24.576874 27094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:40061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=38907
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:24.577183 27094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:24.577437 27094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:24.582841 27128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.582873 27131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.582841 27129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.583045 27094 server_base.cc:1034] running on GCE node
I20260926 05:49:24.583334 27094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:24.583606 27094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:24.584779 27094 hybrid_clock.cc:648] HybridClock initialized: now 1790401764584754 us; error 40 us; skew 500 ppm
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:24.587318 27094 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:49:24.587929 27094 webserver.cc:460] Webserver started at http://127.11.247.1:38907/ using document root <none> and password file <none>
I20260926 05:49:24.588259 27094 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:24.588331 27094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:24.589790 27094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:49:24.590548 27138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.590752 27094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:24.590854 27094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "30b074e08e0445b7be845fd2a1159f74"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "b8322ba523f0debf330ea9f43f7f1ba9"
server_key_iv: "4b07b4b364f3d2c8116829e311069bc4"
server_key_version: "encryptionkey@0"
I20260926 05:49:24.591243 27094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:24.607925 27094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:24.610522 27094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:24.610726 27094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:24.610996 27094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:24.611474 27145 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:49:24.613102 27094 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:49:24.613186 27094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:49:24.613237 27094 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:49:24.614081 27094 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:49:24.614145 27094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:49:24.614259 27145 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap starting.
I20260926 05:49:24.622009 27094 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:40061
I20260926 05:49:24.622110 27252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:40061 every 8 connection(s)
I20260926 05:49:24.622519 27094 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:49:24.625290 27145 log.cc:824] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:24.627857 27145 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:49:24.628175 27145 tablet_bootstrap.cc:492] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Bootstrap complete.
I20260926 05:49:24.629122 27145 ts_tablet_manager.cc:1397] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260926 05:49:24.629851 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27094
I20260926 05:49:24.630132 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:35675
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:42097
--webserver_port=41575
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:49:24.630811 27145 raft_consensus.cc:348] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.631214 27145 raft_consensus.cc:729] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b074e08e0445b7be845fd2a1159f74, State: Initialized, Role: FOLLOWER
I20260926 05:49:24.631376 27145 consensus_queue.cc:260] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:24.631713 27145 ts_tablet_manager.cc:1428] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260926 05:49:24.723629 27257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:24.723894 27257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:24.723938 27257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:24.725261 27257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:24.725344 27257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:24.725402 27257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:49:24.727001 27257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:35675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=41575
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:24.727283 27257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:24.727540 27257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:24.733049 27263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.733044 27266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.733029 27264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.733426 27257 server_base.cc:1034] running on GCE node
I20260926 05:49:24.733714 27257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:24.734027 27257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:24.735219 27257 hybrid_clock.cc:648] HybridClock initialized: now 1790401764735199 us; error 35 us; skew 500 ppm
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:24.738775 27257 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:49:24.739430 27257 webserver.cc:460] Webserver started at http://127.11.247.2:41575/ using document root <none> and password file <none>
I20260926 05:49:24.739671 27257 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:24.739717 27257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:24.741611 27257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:49:24.742333 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.742496 27257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:24.742628 27257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "fa16fa2830964094abe6ae33a15015f5"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "2ec7871d6dae41b58ec86f3eed5938e4"
server_key_iv: "de818e98a849c55cb4331275c948de82"
server_key_version: "encryptionkey@0"
I20260926 05:49:24.743006 27257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:24.755129 27257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:24.757613 27257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:24.757733 27257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:24.758028 27257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:24.758351 27257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:24.758383 27257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.758426 27257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:24.758452 27257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.765713 27257 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35675
I20260926 05:49:24.765760 27386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35675 every 8 connection(s)
I20260926 05:49:24.766184 27257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
W20260926 05:49:24.768006 27255 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:49:24.623346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:58179)
0926 05:49:24.623561 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.623613 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.624301 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.767850 (+143549us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
W20260926 05:49:24.768253 27253 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:42097 (0 consecutive failures): Network error: Failed to ping master at 127.11.247.62:42097: Client connection negotiation failed: client connection to 127.11.247.62:42097: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:49:24.768615 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.768495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:44411)
0926 05:49:24.768557 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.768595 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 05:49:24.768905 27389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:49:24.766970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:44551)
0926 05:49:24.767155 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.767211 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:24.768082 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:24.768266 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260926 05:49:24.768919 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.768831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:44911)
0926 05:49:24.768875 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.768898 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:49:24.769043 27253 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:49:24.769155 27387 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:42097 (0 consecutive failures): Network error: Failed to ping master at 127.11.247.62:42097: Client connection negotiation failed: client connection to 127.11.247.62:42097: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260926 05:49:24.769445 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.769356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:41965)
0926 05:49:24.769396 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.769427 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 05:49:24.769698 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.769630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:45515)
0926 05:49:24.769659 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.769686 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 05:49:24.769804 27387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:49:24.776249 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27257
I20260926 05:49:24.776479 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:44477
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:42097
--webserver_port=41325
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:24.858852 27390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:24.859099 27390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:24.859139 27390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:24.860352 27390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:24.860433 27390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:24.860478 27390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:49:24.862020 27390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:44477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=41325
--tserver_master_addrs=127.11.247.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:24.862334 27390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:24.862617 27390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:24.867926 27396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.868067 27390 server_base.cc:1034] running on GCE node
W20260926 05:49:24.867950 27395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:24.868211 27398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:24.868501 27390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:24.868839 27390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:24.870038 27390 hybrid_clock.cc:648] HybridClock initialized: now 1790401764870016 us; error 34 us; skew 500 ppm
Sep 26 05:49:24 dist-test-slave-21p4 krb5kdc[25954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:24.872691 27390 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:49:24.873282 27390 webserver.cc:460] Webserver started at http://127.11.247.3:41325/ using document root <none> and password file <none>
I20260926 05:49:24.873512 27390 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:24.873584 27390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:24.875025 27390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:24.875667 27405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.875844 27390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:49:24.875945 27390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "8448dbaca7a84251928d864c13a86bff"
format_stamp: "Formatted at 2026-09-26 05:49:23 on dist-test-slave-21p4"
server_key: "c24476daa21f8f176ff08c4eec443306"
server_key_iv: "a2cf18b8c55c32b937998e2693707add"
server_key_version: "encryptionkey@0"
I20260926 05:49:24.876349 27390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:24.891714 27390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:24.893846 27390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:24.894019 27390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:24.894287 27390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:24.894680 27390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:24.894743 27390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.894786 27390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:24.894815 27390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:24.901010 27390 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:44477
I20260926 05:49:24.901054 27518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:44477 every 8 connection(s)
I20260926 05:49:24.901484 27390 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:49:24.902768 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.902236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:57505)
0926 05:49:24.902462 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.902544 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260926 05:49:24.903179 27519 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:42097 (0 consecutive failures): Network error: Failed to ping master at 127.11.247.62:42097: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
I20260926 05:49:24.903474 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.903375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:50757)
0926 05:49:24.903426 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.903454 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:49:24.903761 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:24.903674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:44541)
0926 05:49:24.903720 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:24.903744 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:49:24.903888 27519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:49:24.911621 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27390
I20260926 05:49:25.769722 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:25.769457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:35433)
0926 05:49:25.769612 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:25.769664 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260926 05:49:25.770277 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:25.770102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:33017)
0926 05:49:25.770197 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:25.770229 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
I20260926 05:49:25.904585 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:25.904294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:55435)
0926 05:49:25.904465 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:25.904514 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
I20260926 05:49:26.471760 27526 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:26.471895 27526 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:26.472321 27526 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
Sep 26 05:49:26 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.3@KRBTEST.COM
Sep 26 05:49:26 dist-test-slave-21p4 krb5kdc[25954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.2@KRBTEST.COM
I20260926 05:49:26.484139 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.472543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35675 (local address 127.11.247.1:39905)
0926 05:49:26.472673 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:26.472686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:26.472755 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:26.473280 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:26.473284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:26.473684 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:26.473926 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:26.473935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.475314 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.475315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:26.476073 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:26.476076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.476987 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.476988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:26.477054 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:26.477726 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:26.477731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:26.480104 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:26.482382 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:26.482383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:26.482383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:26.482723 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:26.483017 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:26.483018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:26.483019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:26.483150 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:26.483635 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:26.483637 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:26.483777 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 05:49:26.483874 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 05:49:26.484060 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":53,"thread_start_us":67,"threads_started":1}
I20260926 05:49:26.484299 27530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.472696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:60375 (local address 127.11.247.3:44477)
0926 05:49:26.472917 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:26.472927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:26.472947 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:49:26.473411 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:26.473413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:26.473455 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:26.473613 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:26.474073 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.475098 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.476194 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.476717 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.476859 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:26.479868 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:26.479875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:26.479876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:26.479890 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:26.482026 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:26.482819 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:26.482822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:26.482823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:26.482952 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:26.483229 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:26.483230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:26.483231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:26.483520 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:26.483586 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:26.483999 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 05:49:26.484180 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":52,"threads_started":1}
I20260926 05:49:26.484437 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.472547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44477 (local address 127.11.247.1:60375)
0926 05:49:26.472699 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:26.472714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:26.472780 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:26.473587 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:26.473588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:26.473801 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:26.473926 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:26.473935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.475232 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.475233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:26.476071 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:26.476074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.476880 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.476882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:26.477016 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:26.477686 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:26.477695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:26.479667 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:26.482156 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:26.482170 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:26.482173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:26.482723 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:26.483048 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:26.483049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:26.483050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:26.483151 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:26.483602 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:26.483609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:26.483770 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 05:49:26.483887 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 05:49:26.484159 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":112,"thread_start_us":59,"threads_started":1}
I20260926 05:49:26.484632 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.472669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:39905 (local address 127.11.247.2:35675)
0926 05:49:26.472878 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:26.472883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:26.472909 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:49:26.473028 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:26.473030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:26.473055 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:26.473184 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:26.474085 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.475193 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.476184 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:26.476769 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:26.476889 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:26.480232 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:26.480237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:26.480237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:26.480249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:26.482268 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:26.482819 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:26.482822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:26.482823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:26.482936 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:26.483222 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:26.483223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:26.483223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:26.483528 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:26.483643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:26.484350 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 05:49:26.484511 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":58,"threads_started":1}
W20260926 05:49:26.485383 27140 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
W20260926 05:49:26.485533 27141 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:26.485602 27141 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74; no voters: 8448dbaca7a84251928d864c13a86bff, fa16fa2830964094abe6ae33a15015f5
I20260926 05:49:26.485725 27526 raft_consensus.cc:2726] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:49:26.770459 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.770290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:41011)
0926 05:49:26.770370 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:26.770416 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 05:49:26.771008 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.770751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:48587)
0926 05:49:26.770890 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:26.770943 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260926 05:49:26.905455 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:26.905185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:41569)
0926 05:49:26.905339 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:26.905399 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260926 05:49:27.771384 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:27.771117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:49753)
0926 05:49:27.771290 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:27.771336 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260926 05:49:27.771762 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:27.771564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:48519)
0926 05:49:27.771689 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:27.771720 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260926 05:49:27.906242 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:27.905983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:35309)
0926 05:49:27.906152 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:27.906195 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260926 05:49:28.632014 27536 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:28.632113 27536 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:28.632292 27536 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
W20260926 05:49:28.632754 27140 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
W20260926 05:49:28.632840 27141 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:28.632925 27141 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74; no voters: 8448dbaca7a84251928d864c13a86bff, fa16fa2830964094abe6ae33a15015f5
I20260926 05:49:28.633005 27536 raft_consensus.cc:2726] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:49:28.772413 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:28.772144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:53247)
0926 05:49:28.772313 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:28.772363 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260926 05:49:28.772435 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:28.772161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:45167)
0926 05:49:28.772328 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:28.772364 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260926 05:49:28.907078 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:28.906829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:45563)
0926 05:49:28.906968 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:28.907014 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260926 05:49:29.773283 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:29.773036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:36559)
0926 05:49:29.773200 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:29.773232 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260926 05:49:29.773299 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:29.773046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:57041)
0926 05:49:29.773212 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:29.773256 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260926 05:49:29.907919 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:29.907661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:44981)
0926 05:49:29.907815 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:29.907860 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260926 05:49:30.649570 27543 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:30.649679 27543 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:30.649855 27543 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
W20260926 05:49:30.650266 27141 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
W20260926 05:49:30.650362 27140 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:30.650516 27140 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74; no voters: 8448dbaca7a84251928d864c13a86bff, fa16fa2830964094abe6ae33a15015f5
I20260926 05:49:30.650620 27543 raft_consensus.cc:2726] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:49:30.774190 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:30.773929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:39481)
0926 05:49:30.774105 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:30.774142 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260926 05:49:30.774195 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:30.773929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:33557)
0926 05:49:30.774104 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:30.774153 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260926 05:49:30.908708 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:30.908462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:42701)
0926 05:49:30.908616 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:30.908658 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260926 05:49:31.775081 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:31.774829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:37461)
0926 05:49:31.774978 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:31.775030 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260926 05:49:31.775270 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:31.774911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:43739)
0926 05:49:31.775120 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:31.775171 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":127,"threads_started":1}
I20260926 05:49:31.909533 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:31.909214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:45849)
0926 05:49:31.909390 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:31.909465 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260926 05:49:32.776194 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:32.775844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:42359)
0926 05:49:32.776058 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:32.776129 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260926 05:49:32.776219 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:32.775837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:33897)
0926 05:49:32.776073 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:32.776116 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
I20260926 05:49:32.910547 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:32.910216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:37103)
0926 05:49:32.910402 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:32.910475 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260926 05:49:33.777235 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:33.776902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:39391)
0926 05:49:33.777093 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:33.777164 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260926 05:49:33.777316 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:33.776927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:36883)
0926 05:49:33.777195 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:33.777252 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":136,"threads_started":1}
I20260926 05:49:33.911763 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:33.911395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:58617)
0926 05:49:33.911633 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:33.911699 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
I20260926 05:49:34.778115 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:34.777864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:55195)
0926 05:49:34.778012 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:34.778054 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260926 05:49:34.778124 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:34.777857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:59435)
0926 05:49:34.778048 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:34.778082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260926 05:49:34.912488 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:34.912277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:51549)
0926 05:49:34.912406 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:34.912438 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260926 05:49:35.080490 27559 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:35.080593 27559 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:35.080835 27559 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
W20260926 05:49:35.081247 27140 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
W20260926 05:49:35.081365 27141 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:35.081444 27141 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74; no voters: 8448dbaca7a84251928d864c13a86bff, fa16fa2830964094abe6ae33a15015f5
I20260926 05:49:35.081521 27559 raft_consensus.cc:2726] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:49:35.778970 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:35.778750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:54809)
0926 05:49:35.778899 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:35.778931 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260926 05:49:35.778960 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:35.778728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:43795)
0926 05:49:35.778858 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:35.778900 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260926 05:49:35.913276 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:35.913018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:41869)
0926 05:49:35.913179 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:35.913225 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260926 05:49:36.779927 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:36.779655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:35081)
0926 05:49:36.779832 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:36.779874 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260926 05:49:36.779942 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:36.779637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:59069)
0926 05:49:36.779820 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:36.779887 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260926 05:49:36.914119 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:36.913910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:46343)
0926 05:49:36.914034 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:36.914069 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260926 05:49:37.780850 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:37.780507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:35079)
0926 05:49:37.780748 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:37.780795 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":62,"threads_started":1}
I20260926 05:49:37.780875 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:37.780539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:45859)
0926 05:49:37.780795 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:37.780831 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":196,"threads_started":1}
I20260926 05:49:37.915047 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:37.914805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:44329)
0926 05:49:37.914957 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:37.914996 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260926 05:49:38.781634 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:38.781359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:42753)
0926 05:49:38.781550 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:38.781589 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260926 05:49:38.781634 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:38.781359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:41341)
0926 05:49:38.781536 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:38.781576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260926 05:49:38.915843 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:38.915591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:35157)
0926 05:49:38.915744 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:38.915792 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260926 05:49:39.225509 27572 raft_consensus.cc:482] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:39.225603 27572 raft_consensus.cc:504] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa16fa2830964094abe6ae33a15015f5" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35675 } } peers { permanent_uuid: "8448dbaca7a84251928d864c13a86bff" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44477 } } peers { permanent_uuid: "30b074e08e0445b7be845fd2a1159f74" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 40061 } }
I20260926 05:49:39.225775 27572 leader_election.cc:288] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675), 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477)
W20260926 05:49:39.226145 27140 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8448dbaca7a84251928d864c13a86bff (127.11.247.3:44477): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
W20260926 05:49:39.226219 27141 leader_election.cc:341] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa16fa2830964094abe6ae33a15015f5 (127.11.247.2:35675): Not found: Tablet not found: 870338f3f6554976ae69d53f4a9eb529
I20260926 05:49:39.226293 27141 leader_election.cc:302] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30b074e08e0445b7be845fd2a1159f74; no voters: 8448dbaca7a84251928d864c13a86bff, fa16fa2830964094abe6ae33a15015f5
I20260926 05:49:39.226387 27572 raft_consensus.cc:2726] T 870338f3f6554976ae69d53f4a9eb529 P 30b074e08e0445b7be845fd2a1159f74 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:49:39.782469 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:39.782254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.2:59371)
0926 05:49:39.782385 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:39.782419 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260926 05:49:39.782481 27574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:39.782260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.1:52607)
0926 05:49:39.782398 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:39.782431 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260926 05:49:39.916592 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27094
I20260926 05:49:39.917910 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:39.916493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42097 (local address 127.11.247.3:59759)
0926 05:49:39.916745 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:39.916789 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":67,"threads_started":1}
I20260926 05:49:39.921754 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27257
I20260926 05:49:39.926038 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27390
2026-09-26T05:49:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18385 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:49:39 dist-test-slave-21p4 krb5kdc[27578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:49:39 dist-test-slave-21p4 krb5kdc[27578](info): set up 2 sockets
Sep 26 05:49:39 dist-test-slave-21p4 krb5kdc[27578](info): commencing operation
krb5kdc: starting...
W20260926 05:49:40.970902 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:49:41.099465 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40353
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:41.182986 27591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:41.183203 27591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:41.183225 27591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:41.184486 27591 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:49:41.184535 27591 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:49:41.184548 27591 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:49:41.184559 27591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:49:41.184573 27591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:41.184587 27591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:41.184634 27591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:49:41.184650 27591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:49:41.186151 27591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:41.186415 27591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:41.186700 27591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:41.191922 27596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.191941 27599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.192011 27597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:41.192080 27591 server_base.cc:1034] running on GCE node
I20260926 05:49:41.192483 27591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:41.192824 27591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:41.193980 27591 hybrid_clock.cc:648] HybridClock initialized: now 1790401781193959 us; error 42 us; skew 500 ppm
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:41.201506 27591 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:49:41.202230 27591 webserver.cc:460] Webserver started at http://127.11.247.62:33135/ using document root <none> and password file <none>
I20260926 05:49:41.202477 27591 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:41.202526 27591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:41.202704 27591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:41.203873 27591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "ca3ae9a35a0143038eff4ee31f62dfc0"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "b28dcd406c94a905178e3258361b58ff"
server_key_iv: "fb6afeab89e82c0c31bc238693e58365"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.204269 27591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "ca3ae9a35a0143038eff4ee31f62dfc0"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "b28dcd406c94a905178e3258361b58ff"
server_key_iv: "fb6afeab89e82c0c31bc238693e58365"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.205754 27591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:49:41.206503 27606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.206720 27591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:41.206820 27591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "ca3ae9a35a0143038eff4ee31f62dfc0"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "b28dcd406c94a905178e3258361b58ff"
server_key_iv: "fb6afeab89e82c0c31bc238693e58365"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.206899 27591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:41.218102 27591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:41.220589 27591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:41.220760 27591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:41.225206 27591 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:40353
I20260926 05:49:41.225291 27658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:40353 every 8 connection(s)
I20260926 05:49:41.225628 27591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:49:41.226541 27659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:41.228879 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap starting.
I20260926 05:49:41.229431 27659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:41.229674 27659 log.cc:824] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:41.230281 27659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: No bootstrap required, opened a new log
I20260926 05:49:41.231930 27659 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:49:41.232085 27659 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:41.232110 27659 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3ae9a35a0143038eff4ee31f62dfc0, State: Initialized, Role: FOLLOWER
I20260926 05:49:41.232223 27659 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:49:41.232306 27659 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:49:41.232347 27659 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:49:41.232396 27659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:41.232915 27659 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:49:41.233042 27659 leader_election.cc:302] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3ae9a35a0143038eff4ee31f62dfc0; no voters:
I20260926 05:49:41.233246 27659 leader_election.cc:288] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:49:41.233274 27662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:41.233397 27662 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 LEADER]: Becoming Leader. State: Replica: ca3ae9a35a0143038eff4ee31f62dfc0, State: Running, Role: LEADER
I20260926 05:49:41.233521 27659 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:49:41.233556 27662 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:49:41.234248 27663 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } } }
I20260926 05:49:41.234256 27664 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3ae9a35a0143038eff4ee31f62dfc0. Latest consensus state: current_term: 1 leader_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } } }
I20260926 05:49:41.234359 27663 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:41.234413 27664 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:49:41.234687 27671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:49:41.234953 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27591
I20260926 05:49:41.235069 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:49:41.235273 12252 external_mini_cluster.cc:1428] Setting key 98a7e76a46be832f3da418721c3172d5
I20260926 05:49:41.235980 27671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:49:41.237645 27671 catalog_manager.cc:1347] Generated new cluster ID: f74e23d8228b479ba3f5eb493cf8ba60
I20260926 05:49:41.237699 27671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:41.247406 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.235777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.11.247.62:40353)
0926 05:49:41.236011 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.236021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.236057 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:49:41.236277 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.236285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.236343 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.236491 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:41.237279 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.238782 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.239888 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.240559 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.240659 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.243564 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:41.243570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:41.243570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:41.243584 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:41.245620 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.246010 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.246015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.246015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.246134 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.246336 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.246337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.246337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.246702 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:41.246856 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.247012 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.247169 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260926 05:49:41.257992 27671 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:49:41.258572 27671 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:49:41.264935 27671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Generated new TSK 0
I20260926 05:49:41.265201 27671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:49:41.298120 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40353
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:41.382808 27685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:41.383020 27685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:41.383044 27685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:41.384229 27685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:41.384276 27685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:41.384305 27685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:49:41.385790 27685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:41.386051 27685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:41.386297 27685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:41.391389 27693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.391425 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.391392 27690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:41.391753 27685 server_base.cc:1034] running on GCE node
I20260926 05:49:41.392004 27685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:41.392257 27685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:41.393435 27685 hybrid_clock.cc:648] HybridClock initialized: now 1790401781393422 us; error 31 us; skew 500 ppm
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:41.396209 27685 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:49:41.396689 27685 webserver.cc:460] Webserver started at http://127.11.247.1:43491/ using document root <none> and password file <none>
I20260926 05:49:41.396894 27685 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:41.396935 27685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:41.397027 27685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:41.398032 27685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "431c41bef5494a289bfb4631d9ad325e"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "0fb5583e07fe3d3c4146cb509e8eb7f5"
server_key_iv: "0bffac5515d93c2ddedd36d99a2fc72e"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.398334 27685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "431c41bef5494a289bfb4631d9ad325e"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "0fb5583e07fe3d3c4146cb509e8eb7f5"
server_key_iv: "0bffac5515d93c2ddedd36d99a2fc72e"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.399773 27685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:49:41.400544 27700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.400767 27685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:49:41.400861 27685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "431c41bef5494a289bfb4631d9ad325e"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "0fb5583e07fe3d3c4146cb509e8eb7f5"
server_key_iv: "0bffac5515d93c2ddedd36d99a2fc72e"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.400923 27685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:41.408725 27685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:41.411060 27685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:41.411180 27685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:41.411374 27685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:41.411685 27685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:41.411716 27685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.411773 27685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:41.411799 27685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.417376 27685 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:41155
I20260926 05:49:41.417455 27813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:41155 every 8 connection(s)
I20260926 05:49:41.417775 27685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:49:41.423357 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27685
I20260926 05:49:41.423491 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:49:41.423729 12252 external_mini_cluster.cc:1428] Setting key 259f72142dd417166b6ce17ab4a49ddf
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:41.428483 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.418599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:56453 (local address 127.11.247.62:40353)
0926 05:49:41.418672 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.418675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.418873 (+ 198us) server_negotiation.cc:407] Connection header received
0926 05:49:41.419423 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.419424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.419440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.419491 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:41.420539 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.421073 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.422112 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.422346 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.422425 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.425226 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:41.425227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:41.425228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:41.425238 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:41.426974 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.427386 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.427388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.427388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.427449 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.427693 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.427694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.427694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.427847 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:41.427916 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.428226 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.428330 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:49:41.428548 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.418455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:56453)
0926 05:49:41.418726 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.418777 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.419311 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.419605 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.419609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.420227 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:41.420426 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.420431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.421172 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.421173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.422011 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.422014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.422434 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.422435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.422627 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.423378 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:41.423387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:41.425104 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:41.427059 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:41.427067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:41.427069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:41.427306 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:41.427516 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:41.427517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:41.427517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:41.427628 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:41.427912 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:41.427917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:41.428042 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.428153 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.428377 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
I20260926 05:49:41.429227 27814 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40353
I20260926 05:49:41.429347 27814 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:41.429572 27814 heartbeater.cc:511] Master 127.11.247.62:40353 requested a full tablet report, sending...
I20260926 05:49:41.430151 27623 ts_manager.cc:194] Registered new tserver with Master: 431c41bef5494a289bfb4631d9ad325e (127.11.247.1:41155)
I20260926 05:49:41.430851 27623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:56453
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:49:41.450045 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40353
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:41.533522 27819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:41.533828 27819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:41.533874 27819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:41.535399 27819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:41.535462 27819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:41.535509 27819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:49:41.537062 27819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:41.537279 27819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:41.537472 27819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:41.542979 27825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.542984 27824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.542984 27827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:41.543005 27819 server_base.cc:1034] running on GCE node
I20260926 05:49:41.543447 27819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:41.543790 27819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:41.544983 27819 hybrid_clock.cc:648] HybridClock initialized: now 1790401781544940 us; error 42 us; skew 500 ppm
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:41.547792 27819 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:49:41.548326 27819 webserver.cc:460] Webserver started at http://127.11.247.2:37683/ using document root <none> and password file <none>
I20260926 05:49:41.548578 27819 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:41.548648 27819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:41.548795 27819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:41.549892 27819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "c12dab0c8692449291125efab5719dc2"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "9bb48636f016638d6b93059709feedc4"
server_key_iv: "4d4b972ed160931372a659c0cd84dbd9"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.550274 27819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "c12dab0c8692449291125efab5719dc2"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "9bb48636f016638d6b93059709feedc4"
server_key_iv: "4d4b972ed160931372a659c0cd84dbd9"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.551820 27819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:49:41.552825 27834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.553121 27819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:41.553254 27819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "c12dab0c8692449291125efab5719dc2"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "9bb48636f016638d6b93059709feedc4"
server_key_iv: "4d4b972ed160931372a659c0cd84dbd9"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.553344 27819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:41.565078 27819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:41.567474 27819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:41.567665 27819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:41.567917 27819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:41.568269 27819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:41.568332 27819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.568370 27819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:41.568413 27819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.574677 27819 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:46541
I20260926 05:49:41.574730 27947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:46541 every 8 connection(s)
I20260926 05:49:41.575109 27819 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:41.585335 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27819
I20260926 05:49:41.585448 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:49:41.585677 12252 external_mini_cluster.cc:1428] Setting key b19eac1cda3c49a741b92fbd23d4c7ee
I20260926 05:49:41.586028 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.576007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:34243 (local address 127.11.247.62:40353)
0926 05:49:41.576093 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.576096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.576374 (+ 278us) server_negotiation.cc:407] Connection header received
0926 05:49:41.576875 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.576877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.576888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.576950 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:41.577992 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.578505 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.579622 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.579846 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.579912 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.582396 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:41.582398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:41.582398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:41.582407 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:41.583968 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.584457 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.584459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.584459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.584535 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.584959 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.584961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.584961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.585179 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:41.585285 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.585821 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.585930 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:49:41.586272 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.575963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:34243)
0926 05:49:41.576202 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.576242 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.576763 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.577034 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.577037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.577630 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:41.577832 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.577836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.578648 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.578649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.579524 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.579526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.579954 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.579955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.580114 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.580736 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:41.580744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:41.582280 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:41.584076 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:41.584087 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:41.584089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:41.584352 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:41.584633 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:41.584635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:41.584635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:41.584845 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:41.585275 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:41.585284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:41.585622 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.585797 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.585994 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260926 05:49:41.587073 27948 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40353
I20260926 05:49:41.587209 27948 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:41.587435 27948 heartbeater.cc:511] Master 127.11.247.62:40353 requested a full tablet report, sending...
I20260926 05:49:41.587893 27623 ts_manager.cc:194] Registered new tserver with Master: c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541)
I20260926 05:49:41.588399 27623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:34243
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:49:41.615335 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40353
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:49:41.699867 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:49:41.700141 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:49:41.700192 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:49:41.701462 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:49:41.701546 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:49:41.701596 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:49:41.703255 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:49:41.703540 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:49:41.703785 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:49:41.709306 27961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:41.709416 27953 server_base.cc:1034] running on GCE node
W20260926 05:49:41.709316 27958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:49:41.709306 27959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:49:41.709830 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:49:41.710158 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:49:41.711347 27953 hybrid_clock.cc:648] HybridClock initialized: now 1790401781711337 us; error 45 us; skew 500 ppm
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:49:41.714074 27953 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:49:41.714573 27953 webserver.cc:460] Webserver started at http://127.11.247.3:39805/ using document root <none> and password file <none>
I20260926 05:49:41.714879 27953 fs_manager.cc:359] Metadata directory not provided
I20260926 05:49:41.714926 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:49:41.715104 27953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:49:41.716198 27953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "5f01fe5be374409db59eea82d308e170"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "63b26487ed3991c5c08280e4340a90bd"
server_key_iv: "54ebe9dc61e0a4a13e6bbe2a9f9c9a19"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.716575 27953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "5f01fe5be374409db59eea82d308e170"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "63b26487ed3991c5c08280e4340a90bd"
server_key_iv: "54ebe9dc61e0a4a13e6bbe2a9f9c9a19"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.717986 27953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:49:41.718924 27968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.719136 27953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:49:41.719233 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "5f01fe5be374409db59eea82d308e170"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "63b26487ed3991c5c08280e4340a90bd"
server_key_iv: "54ebe9dc61e0a4a13e6bbe2a9f9c9a19"
server_key_version: "encryptionkey@0"
I20260926 05:49:41.719308 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:49:41.734866 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:49:41.737015 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:49:41.737181 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:49:41.737440 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:49:41.737798 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:49:41.737843 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.737890 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:49:41.737917 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:49:41.744154 27953 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:46251
I20260926 05:49:41.744290 28081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:46251 every 8 connection(s)
I20260926 05:49:41.744590 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:49:41.750968 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 27953
I20260926 05:49:41.751067 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:49:41.751268 12252 external_mini_cluster.cc:1428] Setting key 49984eadc713bbefeaa8aace1e20ba97
Sep 26 05:49:41 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401781, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:49:41.755043 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.745549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:46503 (local address 127.11.247.62:40353)
0926 05:49:41.745613 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.745615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.745920 (+ 305us) server_negotiation.cc:407] Connection header received
0926 05:49:41.746452 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.746453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.746464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.746527 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:41.747646 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.748161 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.749243 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.749448 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.749514 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.752205 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:41.752207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:41.752207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:41.752217 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:41.753557 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.753993 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.753995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.753995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.754051 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.754319 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.754320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.754320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.754444 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:41.754519 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.754858 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.754940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:49:41.755259 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.745540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:46503)
0926 05:49:41.745734 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.745778 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.746334 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.746660 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.746664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.747306 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:49:41.747531 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.747536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.748269 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.748270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.749141 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.749144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.749569 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.749571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.749693 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.750237 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:49:41.750246 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:49:41.752113 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:49:41.753653 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:41.753662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:41.753665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:41.753906 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:41.754124 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:49:41.754125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:49:41.754125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:49:41.754250 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:49:41.754519 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:49:41.754528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:49:41.754720 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.754834 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.755009 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260926 05:49:41.755756 28082 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40353
I20260926 05:49:41.755872 28082 heartbeater.cc:464] Registering TS with master...
I20260926 05:49:41.756072 28082 heartbeater.cc:511] Master 127.11.247.62:40353 requested a full tablet report, sending...
I20260926 05:49:41.756498 27623 ts_manager.cc:194] Registered new tserver with Master: 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
I20260926 05:49:41.756891 27623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:46503
I20260926 05:49:41.763489 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:49:41.771284 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.764903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.11.247.62:40353)
0926 05:49:41.764955 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.764957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.765082 (+ 125us) server_negotiation.cc:407] Connection header received
0926 05:49:41.765150 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.765151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.765160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.765220 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:49:41.765772 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.766369 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.767245 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.767451 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.767516 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.768797 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:49:41.768798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:49:41.768798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:49:41.768805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:49:41.770116 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.770463 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.770463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.770464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.770514 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:49:41.770774 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:49:41.770774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:49:41.770775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:49:41.770925 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:49:41.771007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.771155 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.771216 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:49:41.772919 27623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:49:41.774571 27623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:49:41.781656 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.777442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.11.247.2:46541)
0926 05:49:41.777651 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.777656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.777668 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:49:41.777712 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.777714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.777732 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.777819 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:41.778083 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.778902 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.780396 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.781153 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.781231 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.781292 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.781503 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.781560 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260926 05:49:41.781649 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.777247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41155 (local address 127.0.0.1:59708)
0926 05:49:41.777674 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.777681 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.777708 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.778028 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.778029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.778031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:41.778092 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.778094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.779495 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.779496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.780561 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.780563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.781269 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.781270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.781376 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.781392 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.781454 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.781523 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":72,"thread_start_us":86,"threads_started":1}
I20260926 05:49:41.781817 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.777248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:46251 (local address 127.0.0.1:45946)
0926 05:49:41.777425 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.777466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.777584 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.778036 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.778037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.778038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:41.778094 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.778095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.779492 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.779493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.780617 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.780620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.781534 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.781535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.781595 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.781599 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.781629 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.781673 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":46,"thread_start_us":87,"threads_started":1}
I20260926 05:49:41.781962 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.777687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.11.247.3:46251)
0926 05:49:41.777866 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.777869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.777879 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:49:41.777913 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.777915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.777930 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.778168 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:41.778241 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.779366 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.780713 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.781415 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.781719 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.781792 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.781822 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.781872 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260926 05:49:41.781648 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.777247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46541 (local address 127.0.0.1:44338)
0926 05:49:41.777426 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.777466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.777535 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.777919 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.777922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.777925 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:41.777998 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.778000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.778999 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.779000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.780314 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.780316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.781266 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.781267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.781383 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.781392 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.781466 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.781523 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":24,"thread_start_us":63,"threads_started":1}
I20260926 05:49:41.782578 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.777511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.11.247.1:41155)
0926 05:49:41.777852 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.777858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.777869 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:41.777908 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.777910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.777924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.778168 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:41.778241 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.779389 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.780656 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.781163 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.782318 (+ 1155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.782396 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.782424 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.782481 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":67,"threads_started":1}
I20260926 05:49:41.782750 27882 tablet_service.cc:1467] Processing CreateTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 (DEFAULT_TABLE table=test-table [id=92e76fc4591348a2a3046a203da03448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:41.783134 27882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:41.783185 27748 tablet_service.cc:1467] Processing CreateTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 (DEFAULT_TABLE table=test-table [id=92e76fc4591348a2a3046a203da03448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:41.783520 27748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:41.785975 28099 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Bootstrap starting.
I20260926 05:49:41.786139 28100 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Bootstrap starting.
I20260926 05:49:41.786553 28016 tablet_service.cc:1467] Processing CreateTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 (DEFAULT_TABLE table=test-table [id=92e76fc4591348a2a3046a203da03448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:49:41.786808 28100 tablet_bootstrap.cc:654] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:41.786878 28099 tablet_bootstrap.cc:654] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:41.786926 28016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:49:41.787338 28100 log.cc:824] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:41.787447 28099 log.cc:824] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:41.788172 28100 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: No bootstrap required, opened a new log
I20260926 05:49:41.788213 28099 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: No bootstrap required, opened a new log
I20260926 05:49:41.788285 28100 ts_tablet_manager.cc:1397] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:49:41.788295 28099 ts_tablet_manager.cc:1397] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:49:41.789804 28099 raft_consensus.cc:348] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.789983 28099 raft_consensus.cc:374] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:41.790030 28099 raft_consensus.cc:729] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12dab0c8692449291125efab5719dc2, State: Initialized, Role: FOLLOWER
I20260926 05:49:41.790153 28099 consensus_queue.cc:260] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.790467 27948 heartbeater.cc:503] Master 127.11.247.62:40353 was elected leader, sending a full tablet report...
I20260926 05:49:41.790421 28100 raft_consensus.cc:348] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.790653 28100 raft_consensus.cc:374] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:41.790701 28100 raft_consensus.cc:729] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431c41bef5494a289bfb4631d9ad325e, State: Initialized, Role: FOLLOWER
I20260926 05:49:41.790810 28100 consensus_queue.cc:260] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.790467 28099 ts_tablet_manager.cc:1428] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:49:41.791079 28100 ts_tablet_manager.cc:1428] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:49:41.791115 27814 heartbeater.cc:503] Master 127.11.247.62:40353 was elected leader, sending a full tablet report...
I20260926 05:49:41.791339 28104 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Bootstrap starting.
I20260926 05:49:41.792212 28104 tablet_bootstrap.cc:654] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Neither blocks nor log segments found. Creating new log.
I20260926 05:49:41.792714 28104 log.cc:824] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Log is configured to *not* fsync() on all Append() calls
I20260926 05:49:41.793607 28104 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: No bootstrap required, opened a new log
I20260926 05:49:41.793726 28104 ts_tablet_manager.cc:1397] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:49:41.795400 28104 raft_consensus.cc:348] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.795697 28104 raft_consensus.cc:374] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:49:41.795789 28104 raft_consensus.cc:729] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f01fe5be374409db59eea82d308e170, State: Initialized, Role: FOLLOWER
I20260926 05:49:41.795925 28104 consensus_queue.cc:260] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.796258 28104 ts_tablet_manager.cc:1428] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:49:41.796272 28082 heartbeater.cc:503] Master 127.11.247.62:40353 was elected leader, sending a full tablet report...
W20260926 05:49:41.825747 27949 tablet.cc:2369] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:49:41.883536 28105 raft_consensus.cc:482] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:49:41.883702 28105 raft_consensus.cc:504] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.884100 28105 leader_election.cc:288] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251), c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541)
I20260926 05:49:41.887063 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.884402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:44205 (local address 127.11.247.3:46251)
0926 05:49:41.884459 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.884461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.884472 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:49:41.884519 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.884520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.884532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.884600 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:41.884778 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.885275 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.886233 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.886690 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.886769 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.886779 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.886941 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.886986 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:49:41.887063 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.884306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:46251 (local address 127.11.247.1:44205)
0926 05:49:41.884376 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.884390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.884444 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.884618 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.884620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.884622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:41.884697 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.884699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.885361 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.885362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.886132 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.886135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.886778 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.886779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.886864 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.886868 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.886935 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.886980 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260926 05:49:41.887179 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.884480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:55203 (local address 127.11.247.2:46541)
0926 05:49:41.884524 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.884526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.884639 (+ 113us) server_negotiation.cc:407] Connection header received
0926 05:49:41.884696 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.884697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.884708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.884781 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:49:41.884947 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.885413 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.886387 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.886799 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.886887 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.886895 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.887066 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.887105 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:49:41.887385 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.884358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46541 (local address 127.11.247.1:55203)
0926 05:49:41.884558 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.884572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:49:41.884639 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:49:41.884787 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:49:41.884788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:49:41.884789 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:49:41.884868 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.884870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.885503 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.885504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.886278 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:49:41.886280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.886887 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.886888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:49:41.886967 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.886970 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:49:41.887052 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:49:41.887095 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260926 05:49:41.887480 28036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bdc28cec8b4ff1b7c9ce568f2994b6" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f01fe5be374409db59eea82d308e170" is_pre_election: true
I20260926 05:49:41.887669 28036 raft_consensus.cc:2454] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 0.
I20260926 05:49:41.887751 27902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bdc28cec8b4ff1b7c9ce568f2994b6" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12dab0c8692449291125efab5719dc2" is_pre_election: true
I20260926 05:49:41.887902 27902 raft_consensus.cc:2454] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 0.
I20260926 05:49:41.887940 27702 leader_election.cc:302] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e, 5f01fe5be374409db59eea82d308e170; no voters:
I20260926 05:49:41.888118 28105 raft_consensus.cc:2781] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:49:41.888196 28105 raft_consensus.cc:482] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:49:41.888238 28105 raft_consensus.cc:3037] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:41.888976 28105 raft_consensus.cc:504] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.889142 28105 leader_election.cc:288] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 election: Requested vote from peers 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251), c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541)
I20260926 05:49:41.889307 27902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bdc28cec8b4ff1b7c9ce568f2994b6" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12dab0c8692449291125efab5719dc2"
I20260926 05:49:41.889303 28036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bdc28cec8b4ff1b7c9ce568f2994b6" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f01fe5be374409db59eea82d308e170"
I20260926 05:49:41.889391 28036 raft_consensus.cc:3037] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:41.889391 27902 raft_consensus.cc:3037] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:49:41.890317 28036 raft_consensus.cc:2454] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 1.
I20260926 05:49:41.890317 27902 raft_consensus.cc:2454] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 1.
I20260926 05:49:41.890542 27704 leader_election.cc:302] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e, c12dab0c8692449291125efab5719dc2; no voters:
I20260926 05:49:41.890676 28105 raft_consensus.cc:2781] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:49:41.890841 28105 raft_consensus.cc:686] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 LEADER]: Becoming Leader. State: Replica: 431c41bef5494a289bfb4631d9ad325e, State: Running, Role: LEADER
I20260926 05:49:41.890954 28105 consensus_queue.cc:237] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:49:41.891883 27622 catalog_manager.cc:5495] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e reported cstate change: term changed from 0 to 1, leader changed from <none> to 431c41bef5494a289bfb4631d9ad325e (127.11.247.1). New cstate: current_term: 1 leader_uuid: "431c41bef5494a289bfb4631d9ad325e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260926 05:49:41.898939 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.896505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.11.247.1:41155)
0926 05:49:41.896589 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:49:41.896591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:49:41.896599 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:49:41.896634 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:49:41.896634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:49:41.896653 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:49:41.896724 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:49:41.896809 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.897284 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.898106 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:49:41.898293 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:49:41.898357 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:49:41.898519 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:49:41.898696 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:49:41.898778 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:49:41.898813 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:49:41.898833 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 05:49:41.900926 28036 raft_consensus.cc:1261] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Refusing update from remote peer 431c41bef5494a289bfb4631d9ad325e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:41.901288 27902 raft_consensus.cc:1261] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Refusing update from remote peer 431c41bef5494a289bfb4631d9ad325e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:49:41.901279 28105 consensus_queue.cc:1035] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:41.901619 28105 consensus_queue.cc:1035] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:49:41.904341 28114 mvcc.cc:205] Tried to move back new op lower bound from 7333485698662780928 to 7333485698626252800. Current Snapshot: MvccSnapshot[applied={T|T < 7333485698662780928 or (T in {7333485698662780928})}]
I20260926 05:49:41.905750 28113 mvcc.cc:205] Tried to move back new op lower bound from 7333485698662780928 to 7333485698626252800. Current Snapshot: MvccSnapshot[applied={T|T < 7333485698662780928}]
I20260926 05:49:41.905866 27623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:41.905962 27623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:49:41.906731 27623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Sending DeleteTablet for 3 replicas of tablet e7bdc28cec8b4ff1b7c9ce568f2994b6
I20260926 05:49:41.907076 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27685
I20260926 05:49:41.907111 27882 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:41 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:44338
I20260926 05:49:41.907119 27748 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:41 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:59708
I20260926 05:49:41.907315 28016 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:41 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45946
I20260926 05:49:41.908278 28121 tablet_replica.cc:330] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: stopping tablet replica
I20260926 05:49:41.908402 28121 raft_consensus.cc:2229] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:41.908505 28121 raft_consensus.cc:2258] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:41.909682 28121 ts_tablet_manager.cc:1905] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:41.911715 28121 ts_tablet_manager.cc:1918] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:41.911795 28121 log.cc:1196] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/e7bdc28cec8b4ff1b7c9ce568f2994b6
I20260926 05:49:41.912070 28121 ts_tablet_manager.cc:1939] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P c12dab0c8692449291125efab5719dc2: Deleting consensus metadata
I20260926 05:49:41.912744 27610 catalog_manager.cc:4841] TS c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541): tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 (table test-table [id=92e76fc4591348a2a3046a203da03448]) successfully deleted
W20260926 05:49:41.913708 27607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:41155 (error 108)
I20260926 05:49:41.913848 28122 tablet_replica.cc:330] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: stopping tablet replica
I20260926 05:49:41.913982 28122 raft_consensus.cc:2229] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:49:41.914096 28122 raft_consensus.cc:2258] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:49:41.914263 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:49:41.914112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41155 (local address 127.0.0.1:59720)
0926 05:49:41.914184 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:49:41.914226 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 05:49:41.914393 27607 catalog_manager.cc:4584] TS 431c41bef5494a289bfb4631d9ad325e (127.11.247.1:41155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6: Network error: Client connection negotiation failed: client connection to 127.11.247.1:41155: connect: Connection refused (error 111)
I20260926 05:49:41.914420 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27819
I20260926 05:49:41.914681 28122 ts_tablet_manager.cc:1905] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:49:41.916632 28122 ts_tablet_manager.cc:1918] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:49:41.916697 28122 log.cc:1196] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/e7bdc28cec8b4ff1b7c9ce568f2994b6
I20260926 05:49:41.916958 28122 ts_tablet_manager.cc:1939] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 5f01fe5be374409db59eea82d308e170: Deleting consensus metadata
I20260926 05:49:41.917444 27608 catalog_manager.cc:4841] TS 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 (table test-table [id=92e76fc4591348a2a3046a203da03448]) successfully deleted
I20260926 05:49:41.920172 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27953
I20260926 05:49:41.925051 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 27591
I20260926 05:50:01.930969 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40353
--webserver_interface=127.11.247.62
--webserver_port=33135
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:02.033041 28124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:02.033308 28124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:02.033335 28124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:02.034696 28124 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:50:02.034744 28124 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:50:02.034759 28124 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:50:02.034773 28124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:02.034787 28124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:02.034801 28124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:02.034816 28124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:02.034828 28124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:02.036404 28124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:02.036653 28124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:02.036866 28124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:02.047426 28131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:02.047384 28133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:02.047557 28124 server_base.cc:1034] running on GCE node
W20260926 05:50:02.047387 28130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:02.048166 28124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:02.048552 28124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:02.049783 28124 hybrid_clock.cc:648] HybridClock initialized: now 1790401802049762 us; error 38 us; skew 500 ppm
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:02.058768 28124 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:50:02.059351 28124 webserver.cc:460] Webserver started at http://127.11.247.62:33135/ using document root <none> and password file <none>
I20260926 05:50:02.059665 28124 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:02.059737 28124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:02.061910 28124 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:50:02.062913 28140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.063139 28124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:02.063252 28124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "ca3ae9a35a0143038eff4ee31f62dfc0"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "b28dcd406c94a905178e3258361b58ff"
server_key_iv: "fb6afeab89e82c0c31bc238693e58365"
server_key_version: "encryptionkey@0"
I20260926 05:50:02.063841 28124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:02.074204 28124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:02.076465 28124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:02.076639 28124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:02.082018 28124 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:40353
I20260926 05:50:02.082077 28192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:40353 every 8 connection(s)
I20260926 05:50:02.082559 28124 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:02.084543 28193 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:50:02.085330 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap starting.
I20260926 05:50:02.088945 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28124
I20260926 05:50:02.089287 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:41155
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:40353
--webserver_port=43491
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:50:02.098336 28193 log.cc:824] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:02.101097 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:50:02.101464 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap complete.
I20260926 05:50:02.104311 28193 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:02.104619 28193 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3ae9a35a0143038eff4ee31f62dfc0, State: Initialized, Role: FOLLOWER
I20260926 05:50:02.104858 28193 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:02.104974 28193 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:02.105038 28193 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:02.105093 28193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:50:02.106468 28193 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:02.106630 28193 leader_election.cc:302] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3ae9a35a0143038eff4ee31f62dfc0; no voters:
I20260926 05:50:02.106900 28193 leader_election.cc:288] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:50:02.106942 28197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:50:02.107081 28197 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 LEADER]: Becoming Leader. State: Replica: ca3ae9a35a0143038eff4ee31f62dfc0, State: Running, Role: LEADER
I20260926 05:50:02.107246 28193 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:02.107246 28197 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:02.107712 28198 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } } }
I20260926 05:50:02.107805 28198 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:02.108208 28199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3ae9a35a0143038eff4ee31f62dfc0. Latest consensus state: current_term: 2 leader_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } } }
I20260926 05:50:02.108302 28199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:02.108635 28203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:02.109741 28203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:02.109900 28203 catalog_manager.cc:1259] Loaded cluster ID: f74e23d8228b479ba3f5eb493cf8ba60
I20260926 05:50:02.109966 28203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:02.111358 28203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:02.111672 28203 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Loaded TSK: 0
I20260926 05:50:02.125190 28203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Generated new TSK 1
I20260926 05:50:02.125358 28203 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:50:02.190912 28195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:02.191190 28195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:02.191220 28195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:02.192656 28195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:02.192723 28195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:02.192752 28195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:02.194460 28195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:41155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=43491
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:02.194830 28195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:02.195180 28195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:02.203183 28224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:02.203166 28222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:02.203198 28195 server_base.cc:1034] running on GCE node
W20260926 05:50:02.203166 28221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:02.203738 28195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:02.204087 28195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:02.205273 28195 hybrid_clock.cc:648] HybridClock initialized: now 1790401802205258 us; error 34 us; skew 500 ppm
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:02.209007 28195 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:50:02.209664 28195 webserver.cc:460] Webserver started at http://127.11.247.1:43491/ using document root <none> and password file <none>
I20260926 05:50:02.209981 28195 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:02.210078 28195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:02.211915 28195 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:02.212713 28231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.213014 28195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:02.213130 28195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "431c41bef5494a289bfb4631d9ad325e"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "0fb5583e07fe3d3c4146cb509e8eb7f5"
server_key_iv: "0bffac5515d93c2ddedd36d99a2fc72e"
server_key_version: "encryptionkey@0"
I20260926 05:50:02.213727 28195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:02.233314 28195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:02.236732 28195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:02.236930 28195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:02.237274 28195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:02.237982 28238 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:50:02.240283 28195 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:50:02.240356 28195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260926 05:50:02.240410 28195 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:50:02.241456 28195 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:50:02.241508 28195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:50:02.241650 28238 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Bootstrap starting.
I20260926 05:50:02.252167 28195 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:41155
I20260926 05:50:02.252259 28345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:41155 every 8 connection(s)
I20260926 05:50:02.253101 28195 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:02.256417 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28195
I20260926 05:50:02.256796 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:46541
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:40353
--webserver_port=37683
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:50:02.259960 28238 log.cc:824] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:02.264040 28238 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:50:02.264472 28238 tablet_bootstrap.cc:492] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Bootstrap complete.
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:02.265879 28238 ts_tablet_manager.cc:1397] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s
I20260926 05:50:02.268821 28238 raft_consensus.cc:348] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.269310 28238 raft_consensus.cc:729] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431c41bef5494a289bfb4631d9ad325e, State: Initialized, Role: FOLLOWER
I20260926 05:50:02.269558 28238 consensus_queue.cc:260] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.269975 28238 ts_tablet_manager.cc:1428] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 05:50:02.270471 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.254233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:60669)
0926 05:50:02.254555 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.254662 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.255499 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.256463 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.256468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.257082 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:02.257576 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.257587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.259913 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.259915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.261334 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.261339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.262453 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.262455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.262666 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.263429 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:02.263442 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:02.265569 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:02.268380 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:02.268387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:02.268390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:02.268705 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:02.269035 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:02.269035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:02.269036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:02.269160 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:02.269707 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:02.269713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:02.269910 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.270025 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.270241 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":157,"threads_started":1}
I20260926 05:50:02.270738 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.254373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:60669 (local address 127.11.247.62:40353)
0926 05:50:02.254827 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.254850 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.254952 (+ 102us) server_negotiation.cc:407] Connection header received
0926 05:50:02.255884 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.255899 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.255988 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.256227 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:02.257883 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.259700 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.261470 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.262177 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.262300 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.265714 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:02.265720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:02.265721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:02.265737 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:02.268258 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.268806 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.268809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.268810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.268955 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.269232 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.269233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.269234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.269588 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:02.269676 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.270338 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.270507 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":112,"threads_started":1}
I20260926 05:50:02.272326 28346 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40353
I20260926 05:50:02.272497 28346 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:02.272715 28346 heartbeater.cc:511] Master 127.11.247.62:40353 requested a full tablet report, sending...
I20260926 05:50:02.273643 28157 ts_manager.cc:194] Registered new tserver with Master: 431c41bef5494a289bfb4631d9ad325e (127.11.247.1:41155)
I20260926 05:50:02.275172 28157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:60669
I20260926 05:50:02.276271 28346 heartbeater.cc:503] Master 127.11.247.62:40353 was elected leader, sending a full tablet report...
I20260926 05:50:02.280365 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.274552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41155 (local address 127.0.0.1:45768)
0926 05:50:02.276084 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.276125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.276210 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.276560 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.276564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.276566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:02.276648 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.276651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.278012 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.278012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.278975 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.278979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.280078 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.280079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.280156 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.280167 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.280239 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.280293 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":141,"threads_started":1}
I20260926 05:50:02.280406 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.275613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.11.247.1:41155)
0926 05:50:02.276353 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.276357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.276366 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:02.276403 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.276405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.276422 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.276524 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:02.276762 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.277904 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.279086 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.279968 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.280058 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.280137 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.280271 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.280323 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":130,"threads_started":1}
I20260926 05:50:02.280897 28280 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:49:41 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260926 05:50:02.281522 28355 tablet_replica.cc:330] stopping tablet replica
I20260926 05:50:02.281648 28355 raft_consensus.cc:2229] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:02.281723 28355 raft_consensus.cc:2258] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:02.282147 28355 ts_tablet_manager.cc:1905] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:02.285167 28355 ts_tablet_manager.cc:1918] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:02.285256 28355 log.cc:1196] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal/wals/e7bdc28cec8b4ff1b7c9ce568f2994b6
I20260926 05:50:02.285497 28355 ts_tablet_manager.cc:1939] T e7bdc28cec8b4ff1b7c9ce568f2994b6 P 431c41bef5494a289bfb4631d9ad325e: Deleting consensus metadata
I20260926 05:50:02.286211 28141 catalog_manager.cc:4841] TS 431c41bef5494a289bfb4631d9ad325e (127.11.247.1:41155): tablet e7bdc28cec8b4ff1b7c9ce568f2994b6 (table test-table [id=92e76fc4591348a2a3046a203da03448]) successfully deleted
W20260926 05:50:02.365010 28350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:02.365252 28350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:02.365278 28350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:02.366526 28350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:02.366570 28350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:02.366633 28350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:02.368227 28350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:46541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=37683
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:02.368525 28350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:02.368786 28350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:02.375129 28361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:02.375288 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:02.375500 28363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:02.377457 28350 server_base.cc:1034] running on GCE node
I20260926 05:50:02.377704 28350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:02.377990 28350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:02.379182 28350 hybrid_clock.cc:648] HybridClock initialized: now 1790401802379176 us; error 38 us; skew 500 ppm
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:02.382202 28350 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:50:02.382835 28350 webserver.cc:460] Webserver started at http://127.11.247.2:37683/ using document root <none> and password file <none>
I20260926 05:50:02.383114 28350 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:02.383208 28350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:02.384845 28350 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:02.385874 28370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.386160 28350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:02.386276 28350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "c12dab0c8692449291125efab5719dc2"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "9bb48636f016638d6b93059709feedc4"
server_key_iv: "4d4b972ed160931372a659c0cd84dbd9"
server_key_version: "encryptionkey@0"
I20260926 05:50:02.386871 28350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:02.397917 28350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:02.400295 28350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:02.400496 28350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:02.400774 28350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:02.401134 28350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:02.401190 28350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.401232 28350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:02.401261 28350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.409152 28350 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:46541
I20260926 05:50:02.409193 28483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:46541 every 8 connection(s)
I20260926 05:50:02.409626 28350 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:02.417089 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28350
I20260926 05:50:02.417409 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:46251
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:40353
--webserver_port=39805
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:50:02.420718 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.410509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:52679 (local address 127.11.247.62:40353)
0926 05:50:02.410627 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.410630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.410962 (+ 332us) server_negotiation.cc:407] Connection header received
0926 05:50:02.411617 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.411619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.411644 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.411704 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:02.412830 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.413425 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.414537 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.414841 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.414926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.417321 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:02.417323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:02.417324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:02.417334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:02.419117 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.419567 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.419569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.419570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.419651 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.419936 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.419937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.419938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.420106 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:02.420183 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.420468 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.420592 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:50:02.420833 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.410519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:52679)
0926 05:50:02.410805 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.410848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.411484 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.411794 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.411798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.412469 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:02.412712 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.412717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.413534 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.413535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.414438 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.414440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.414935 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.414936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.415082 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.415603 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:02.415613 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:02.417149 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:02.419228 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:02.419236 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:02.419238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:02.419484 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:02.419746 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:02.419746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:02.419747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:02.419869 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:02.420176 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:02.420182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:02.420319 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.420421 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.420564 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":200,"threads_started":1}
I20260926 05:50:02.421425 28484 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40353
I20260926 05:50:02.421557 28484 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:02.421766 28484 heartbeater.cc:511] Master 127.11.247.62:40353 requested a full tablet report, sending...
I20260926 05:50:02.422262 28157 ts_manager.cc:194] Registered new tserver with Master: c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541)
I20260926 05:50:02.422735 28157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:52679
W20260926 05:50:02.516947 28487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:02.517268 28487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:02.517313 28487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:02.518798 28487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:02.518893 28487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:02.518950 28487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:02.520736 28487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:46251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=39805
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:02.521068 28487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:02.521360 28487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:02.527652 28492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:02.527647 28495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:02.527647 28493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:02.528020 28487 server_base.cc:1034] running on GCE node
I20260926 05:50:02.528247 28487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:02.528541 28487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:02.529717 28487 hybrid_clock.cc:648] HybridClock initialized: now 1790401802529690 us; error 42 us; skew 500 ppm
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:02.532945 28487 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:50:02.533803 28487 webserver.cc:460] Webserver started at http://127.11.247.3:39805/ using document root <none> and password file <none>
I20260926 05:50:02.534169 28487 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:02.534237 28487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:02.535841 28487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:02.536499 28502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.536705 28487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:50:02.536803 28487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "5f01fe5be374409db59eea82d308e170"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "63b26487ed3991c5c08280e4340a90bd"
server_key_iv: "54ebe9dc61e0a4a13e6bbe2a9f9c9a19"
server_key_version: "encryptionkey@0"
I20260926 05:50:02.537243 28487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:02.572554 28487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:02.575126 28487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:02.575291 28487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:02.575639 28487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:02.576025 28487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:02.576061 28487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.576090 28487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:02.576103 28487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:02.583500 28487 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:46251
I20260926 05:50:02.583563 28615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:46251 every 8 connection(s)
I20260926 05:50:02.583967 28487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:50:02.585637 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28487
Sep 26 05:50:02 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401802, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:02.595829 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.586229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.11.247.62:40353)
0926 05:50:02.586429 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.586430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.586670 (+ 240us) server_negotiation.cc:407] Connection header received
0926 05:50:02.586783 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.586784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.586796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.586887 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:02.588037 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.588963 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.590241 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.590494 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.590568 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.592319 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:02.592321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:02.592321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:02.592328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:02.594386 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.594799 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.594800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.594801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.594881 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.595103 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.595104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.595104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.595238 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:02.595506 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.595592 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.595688 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260926 05:50:02.596611 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.584892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:56853 (local address 127.11.247.62:40353)
0926 05:50:02.584975 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.584978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.585226 (+ 248us) server_negotiation.cc:407] Connection header received
0926 05:50:02.586237 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.586238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.586252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.586321 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:02.588037 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.588963 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.590524 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.590799 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.590850 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.593531 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:02.593533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:02.593533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:02.593542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:02.595050 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.595547 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.595549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.595549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.595615 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.595878 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.595879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.595880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.596027 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:02.596113 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.596458 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.596528 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":42}
I20260926 05:50:02.596941 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.584841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:56853)
0926 05:50:02.585064 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.585115 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.586039 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.586434 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.586440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.587524 (+ 1084us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:02.587923 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.587931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.589092 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.589093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.590432 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.590435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.590885 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.590886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.591071 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.591555 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:02.591566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:02.593368 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:02.595179 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:02.595186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:02.595189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:02.595442 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:02.595690 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:02.595691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:02.595692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:02.595807 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:02.596105 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:02.596114 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:02.596306 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.596445 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.596655 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":124,"threads_started":1}
I20260926 05:50:02.597404 28616 heartbeater.cc:347] Connected to a master server at 127.11.247.62:40353
I20260926 05:50:02.597512 28616 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:02.597735 28616 heartbeater.cc:511] Master 127.11.247.62:40353 requested a full tablet report, sending...
I20260926 05:50:02.598244 28157 ts_manager.cc:194] Registered new tserver with Master: 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
I20260926 05:50:02.598891 28157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:56853
I20260926 05:50:02.607728 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:02.611456 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.608379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.11.247.62:40353)
0926 05:50:02.608429 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.608431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.608656 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:50:02.608753 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.608754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.608814 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.608891 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:02.609145 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.609675 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.610542 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.610824 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.610908 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.611145 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:02.611208 (+ 63us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 05:50:02.611378 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51254: token expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:50:02.611521 28349 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51254: token expired
W20260926 05:50:02.611704 28087 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.247.62:40353)
I20260926 05:50:02.617740 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.611998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.11.247.62:40353)
0926 05:50:02.612050 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.612052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.612061 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:02.612084 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.612085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.612095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.612148 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:02.612680 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.613161 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.613848 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.614044 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.614099 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.615410 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:02.615411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:02.615412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:02.615419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:02.616684 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.617007 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.617008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.617009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.617065 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:02.617238 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:02.617238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:02.617239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:02.617378 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:02.617461 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.617562 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.617625 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:50:02.619093 28157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:02.621227 28157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:02.624060 28280 tablet_service.cc:1467] Processing CreateTablet for tablet f8d9610f584c4fb58b6528fbee4a845d (DEFAULT_TABLE table=test-table [id=98b75ea7b4ea43adb10c2320cdcfa86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:02.624320 28280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d9610f584c4fb58b6528fbee4a845d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:02.626070 28238 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Bootstrap starting.
I20260926 05:50:02.626506 28238 tablet_bootstrap.cc:654] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:02.627194 28238 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: No bootstrap required, opened a new log
I20260926 05:50:02.627251 28238 ts_tablet_manager.cc:1397] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:02.627381 28238 raft_consensus.cc:348] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.627473 28238 raft_consensus.cc:374] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:02.627575 28238 raft_consensus.cc:729] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431c41bef5494a289bfb4631d9ad325e, State: Initialized, Role: FOLLOWER
I20260926 05:50:02.627625 28238 consensus_queue.cc:260] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.627722 28238 ts_tablet_manager.cc:1428] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 05:50:02.627990 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.623696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46541 (local address 127.0.0.1:60178)
0926 05:50:02.623755 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.623769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.623811 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.624368 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.624370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.624373 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:02.624437 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.624439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.625459 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.625460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.626605 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.626609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.627731 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.627732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.627810 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.627816 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.627854 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.627898 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:50:02.628465 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.623920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.11.247.2:46541)
0926 05:50:02.624136 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.624142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.624156 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:50:02.624217 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.624219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.624239 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.624361 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:02.624594 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.625361 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.626736 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.627588 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.628116 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.628214 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.628246 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.628300 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":132,"threads_started":1}
I20260926 05:50:02.629351 28418 tablet_service.cc:1467] Processing CreateTablet for tablet f8d9610f584c4fb58b6528fbee4a845d (DEFAULT_TABLE table=test-table [id=98b75ea7b4ea43adb10c2320cdcfa86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:02.629766 28418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d9610f584c4fb58b6528fbee4a845d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:02.630270 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.623763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:46251 (local address 127.0.0.1:37470)
0926 05:50:02.624177 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.624188 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.624244 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.626363 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.626364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.626367 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:02.626427 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.626428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.627848 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.627850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.628979 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.628981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.630009 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.630010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.630076 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.630080 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.630142 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.630196 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":100,"threads_started":1}
I20260926 05:50:02.630527 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.624034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.11.247.3:46251)
0926 05:50:02.626186 (+ 2152us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.626191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.626203 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:50:02.626249 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.626251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.626269 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.626364 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:02.626622 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.627743 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.629064 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.629820 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.629893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.629953 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.630187 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.630239 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2105,"thread_start_us":111,"threads_started":1}
I20260926 05:50:02.631281 28550 tablet_service.cc:1467] Processing CreateTablet for tablet f8d9610f584c4fb58b6528fbee4a845d (DEFAULT_TABLE table=test-table [id=98b75ea7b4ea43adb10c2320cdcfa86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:02.631631 28550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d9610f584c4fb58b6528fbee4a845d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:02.634335 28626 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Bootstrap starting.
I20260926 05:50:02.635058 28626 tablet_bootstrap.cc:654] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:02.635409 28627 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Bootstrap starting.
I20260926 05:50:02.635852 28626 log.cc:824] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:02.636412 28627 tablet_bootstrap.cc:654] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:02.636863 28626 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: No bootstrap required, opened a new log
I20260926 05:50:02.636945 28626 ts_tablet_manager.cc:1397] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:50:02.637276 28627 log.cc:824] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:02.638163 28627 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: No bootstrap required, opened a new log
I20260926 05:50:02.638233 28627 ts_tablet_manager.cc:1397] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:50:02.639530 28626 raft_consensus.cc:348] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.639750 28626 raft_consensus.cc:374] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:02.639819 28626 raft_consensus.cc:729] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12dab0c8692449291125efab5719dc2, State: Initialized, Role: FOLLOWER
I20260926 05:50:02.639998 28626 consensus_queue.cc:260] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.640338 28626 ts_tablet_manager.cc:1428] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:50:02.640388 28484 heartbeater.cc:503] Master 127.11.247.62:40353 was elected leader, sending a full tablet report...
I20260926 05:50:02.640590 28627 raft_consensus.cc:348] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.640904 28627 raft_consensus.cc:374] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:02.640946 28627 raft_consensus.cc:729] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f01fe5be374409db59eea82d308e170, State: Initialized, Role: FOLLOWER
I20260926 05:50:02.641067 28627 consensus_queue.cc:260] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.641373 28616 heartbeater.cc:503] Master 127.11.247.62:40353 was elected leader, sending a full tablet report...
I20260926 05:50:02.641374 28627 ts_tablet_manager.cc:1428] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 05:50:02.660279 28485 tablet.cc:2369] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:50:02.716912 28352 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:02.717106 28352 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.717622 28352 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
I20260926 05:50:02.721293 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.717909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:42549 (local address 127.11.247.2:46541)
0926 05:50:02.717967 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.717970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.717981 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:50:02.718022 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.718023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.718036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.718115 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:02.718316 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.719007 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.720311 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.720923 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.720984 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.720994 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.721184 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.721226 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:50:02.721319 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.717805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46541 (local address 127.11.247.1:42549)
0926 05:50:02.717871 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.717886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.717931 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.718140 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.718141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.718143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:02.718224 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.718228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.719122 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.719124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.720223 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.720225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.721013 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.721014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.721092 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.721097 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.721161 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.721209 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 05:50:02.721809 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.717820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:46251 (local address 127.11.247.1:54723)
0926 05:50:02.718057 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.718067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:02.718113 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:02.718314 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:02.718315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:02.718317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:02.718814 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.718817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.719730 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.719730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.720741 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:02.720744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.721553 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.721554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:02.721617 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.721621 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:02.721676 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:50:02.721717 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260926 05:50:02.721817 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.718103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:54723 (local address 127.11.247.3:46251)
0926 05:50:02.718155 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.718158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.718165 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:50:02.718196 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.718199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.718214 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.718303 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:02.718910 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.719640 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.720863 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.721463 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.721550 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.721562 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.721687 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.721748 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:50:02.721849 28438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d9610f584c4fb58b6528fbee4a845d" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12dab0c8692449291125efab5719dc2" is_pre_election: true
I20260926 05:50:02.722072 28438 raft_consensus.cc:2454] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 0.
I20260926 05:50:02.722303 28570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d9610f584c4fb58b6528fbee4a845d" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f01fe5be374409db59eea82d308e170" is_pre_election: true
I20260926 05:50:02.722388 28235 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e, c12dab0c8692449291125efab5719dc2; no voters:
I20260926 05:50:02.722486 28570 raft_consensus.cc:2454] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 0.
I20260926 05:50:02.722592 28352 raft_consensus.cc:2781] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:02.722692 28352 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:02.722719 28352 raft_consensus.cc:3037] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:02.723604 28352 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.723764 28352 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 election: Requested vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
I20260926 05:50:02.723949 28438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d9610f584c4fb58b6528fbee4a845d" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12dab0c8692449291125efab5719dc2"
I20260926 05:50:02.723981 28570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d9610f584c4fb58b6528fbee4a845d" candidate_uuid: "431c41bef5494a289bfb4631d9ad325e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f01fe5be374409db59eea82d308e170"
I20260926 05:50:02.724064 28570 raft_consensus.cc:3037] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:02.724064 28438 raft_consensus.cc:3037] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:02.725356 28570 raft_consensus.cc:2454] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 1.
I20260926 05:50:02.725350 28438 raft_consensus.cc:2454] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431c41bef5494a289bfb4631d9ad325e in term 1.
I20260926 05:50:02.725593 28235 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e, c12dab0c8692449291125efab5719dc2; no voters:
I20260926 05:50:02.725701 28352 raft_consensus.cc:2781] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:02.725889 28352 raft_consensus.cc:686] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 LEADER]: Becoming Leader. State: Replica: 431c41bef5494a289bfb4631d9ad325e, State: Running, Role: LEADER
I20260926 05:50:02.725992 28352 consensus_queue.cc:237] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:02.727135 28157 catalog_manager.cc:5495] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e reported cstate change: term changed from 0 to 1, leader changed from <none> to 431c41bef5494a289bfb4631d9ad325e (127.11.247.1). New cstate: current_term: 1 leader_uuid: "431c41bef5494a289bfb4631d9ad325e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } health_report { overall_health: HEALTHY } } }
I20260926 05:50:02.746372 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.743681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.11.247.1:41155)
0926 05:50:02.743760 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:02.743763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:02.743774 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:50:02.743878 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:02.743879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:02.743901 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:02.743993 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:02.744156 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.744695 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.745502 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:02.745711 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:02.745786 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:02.745976 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:02.746103 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:02.746188 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:02.746240 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 05:50:02.746268 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:50:02.748526 28570 raft_consensus.cc:1261] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Refusing update from remote peer 431c41bef5494a289bfb4631d9ad325e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:02.748528 28438 raft_consensus.cc:1261] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Refusing update from remote peer 431c41bef5494a289bfb4631d9ad325e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:02.748978 28633 consensus_queue.cc:1035] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:02.749118 28352 consensus_queue.cc:1035] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:02.753073 28637 mvcc.cc:205] Tried to move back new op lower bound from 7333485784054349824 to 7333485783966580736. Current Snapshot: MvccSnapshot[applied={T|T < 7333485784054349824}]
I20260926 05:50:02.753284 28636 mvcc.cc:205] Tried to move back new op lower bound from 7333485784054349824 to 7333485783966580736. Current Snapshot: MvccSnapshot[applied={T|T < 7333485784054349824}]
I20260926 05:50:02.754773 28157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:02.754880 28157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:02.755784 28157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Sending DeleteTablet for 3 replicas of tablet f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:02.756129 28418 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d9610f584c4fb58b6528fbee4a845d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:02 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:60178
I20260926 05:50:02.756174 28280 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d9610f584c4fb58b6528fbee4a845d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:02 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260926 05:50:02.756222 28550 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d9610f584c4fb58b6528fbee4a845d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:02 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:37470
I20260926 05:50:02.756276 28355 tablet_replica.cc:330] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: stopping tablet replica
I20260926 05:50:02.756326 28355 raft_consensus.cc:2229] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:50:02.756431 28355 raft_consensus.cc:2258] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:02.756536 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 28195
I20260926 05:50:02.756580 28646 tablet_replica.cc:330] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: stopping tablet replica
I20260926 05:50:02.757252 28646 raft_consensus.cc:2229] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:02.757380 28646 raft_consensus.cc:2258] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:02.757822 28646 ts_tablet_manager.cc:1905] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:02.759637 28646 ts_tablet_manager.cc:1918] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:02.759734 28646 log.cc:1196] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:02.760003 28646 ts_tablet_manager.cc:1939] T f8d9610f584c4fb58b6528fbee4a845d P 5f01fe5be374409db59eea82d308e170: Deleting consensus metadata
I20260926 05:50:02.760462 28142 catalog_manager.cc:4841] TS 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): tablet f8d9610f584c4fb58b6528fbee4a845d (table test-table [id=98b75ea7b4ea43adb10c2320cdcfa86e]) successfully deleted
I20260926 05:50:02.762734 28645 tablet_replica.cc:330] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: stopping tablet replica
I20260926 05:50:02.762892 28645 raft_consensus.cc:2229] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:02.763038 28645 raft_consensus.cc:2258] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:02.763530 28645 ts_tablet_manager.cc:1905] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:50:02.764099 28141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:41155 (error 108)
I20260926 05:50:02.764472 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 28350
I20260926 05:50:02.764717 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.764516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41155 (local address 127.0.0.1:45798)
0926 05:50:02.764623 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.764672 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 05:50:02.764957 28141 catalog_manager.cc:4584] TS 431c41bef5494a289bfb4631d9ad325e (127.11.247.1:41155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8d9610f584c4fb58b6528fbee4a845d: Network error: Client connection negotiation failed: client connection to 127.11.247.1:41155: connect: Connection refused (error 111)
I20260926 05:50:02.771389 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:02.771182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46541 (local address 127.0.0.1:60194)
0926 05:50:02.771297 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:02.771340 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.2:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 05:50:02.771373 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 28487
I20260926 05:50:02.777607 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 28124
I20260926 05:50:22.784644 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40353
--webserver_interface=127.11.247.62
--webserver_port=33135
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:22.880055 28648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:22.880343 28648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:22.880370 28648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:22.881825 28648 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:50:22.881879 28648 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:50:22.881893 28648 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:50:22.881906 28648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:22.881918 28648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:22.881932 28648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:22.881944 28648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:22.881956 28648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:22.883906 28648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:22.884209 28648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:22.884456 28648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:22.890828 28654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:22.890930 28653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:22.890857 28656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:22.890895 28648 server_base.cc:1034] running on GCE node
I20260926 05:50:22.891387 28648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:22.891733 28648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:22.892915 28648 hybrid_clock.cc:648] HybridClock initialized: now 1790401822892886 us; error 41 us; skew 500 ppm
Sep 26 05:50:22 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401822, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:22.900993 28648 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:50:22.901707 28648 webserver.cc:460] Webserver started at http://127.11.247.62:33135/ using document root <none> and password file <none>
I20260926 05:50:22.902029 28648 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:22.902083 28648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:22.903868 28648 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:22.904750 28663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:22.904990 28648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:22.905107 28648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "ca3ae9a35a0143038eff4ee31f62dfc0"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "b28dcd406c94a905178e3258361b58ff"
server_key_iv: "fb6afeab89e82c0c31bc238693e58365"
server_key_version: "encryptionkey@0"
I20260926 05:50:22.905601 28648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:22.920034 28648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:22.922806 28648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:22.923007 28648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:22.928400 28648 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:40353
I20260926 05:50:22.928455 28715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:40353 every 8 connection(s)
I20260926 05:50:22.928964 28648 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:22.930896 28716 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:50:22.931639 28716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap starting.
I20260926 05:50:22.931649 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28648
I20260926 05:50:22.931977 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:41155
--local_ip_for_outbound_sockets=127.11.247.1
--tserver_master_addrs=127.11.247.62:40353
--webserver_port=43491
--webserver_interface=127.11.247.1
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:50:22.945829 28716 log.cc:824] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:22.949610 28716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:50:22.950078 28716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Bootstrap complete.
I20260926 05:50:22.953895 28716 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:22.954303 28716 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3ae9a35a0143038eff4ee31f62dfc0, State: Initialized, Role: FOLLOWER
I20260926 05:50:22.954509 28716 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:22.954675 28716 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:22.954749 28716 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:22.954814 28716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:50:22.956887 28716 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:22.957051 28716 leader_election.cc:302] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3ae9a35a0143038eff4ee31f62dfc0; no voters:
I20260926 05:50:22.957273 28716 leader_election.cc:288] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 05:50:22.957378 28720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:50:22.957593 28720 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [term 3 LEADER]: Becoming Leader. State: Replica: ca3ae9a35a0143038eff4ee31f62dfc0, State: Running, Role: LEADER
I20260926 05:50:22.957602 28716 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:22.957814 28720 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } }
I20260926 05:50:22.958303 28721 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } } }
I20260926 05:50:22.958436 28721 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:22.958739 28729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:22.958310 28722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3ae9a35a0143038eff4ee31f62dfc0. Latest consensus state: current_term: 3 leader_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3ae9a35a0143038eff4ee31f62dfc0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 40353 } } }
I20260926 05:50:22.959286 28722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:22.959843 28729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:22.960004 28729 catalog_manager.cc:1259] Loaded cluster ID: f74e23d8228b479ba3f5eb493cf8ba60
I20260926 05:50:22.960240 28729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:22.961757 28729 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:22.961993 28729 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Loaded TSK: 0
I20260926 05:50:22.962046 28729 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca3ae9a35a0143038eff4ee31f62dfc0: Loaded TSK: 1
F20260926 05:50:22.962337 28729 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe5b1ab69ad google::LogMessage::Fail() at ??:0
@ 0x7fe5b1ab988a google::LogMessage::SendToLog() at ??:0
@ 0x7fe5b1ab64de google::LogMessage::Flush() at ??:0
@ 0x7fe5b1ab7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5b4223104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe5b4220d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe5b9a101cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe5b9a1048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe5b9a10594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe5b99d6c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe5b99e0ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe5b39840d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe5b3979954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe5b47dd6db start_thread at ??:0
@ 0x7fe5b05b461f clone at ??:0
W20260926 05:50:23.060379 28718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:23.060652 28718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:23.060683 28718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:23.061998 28718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:23.062047 28718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:23.062078 28718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:23.063905 28718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:41155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=43491
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:23.064229 28718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:23.064508 28718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:23.070302 28760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:23.070302 28757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:23.070425 28718 server_base.cc:1034] running on GCE node
W20260926 05:50:23.070640 28758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:23.070854 28718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:23.071115 28718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:23.072307 28718 hybrid_clock.cc:648] HybridClock initialized: now 1790401823072272 us; error 52 us; skew 500 ppm
Sep 26 05:50:23 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401823, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:23.075596 28718 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:50:23.076236 28718 webserver.cc:460] Webserver started at http://127.11.247.1:43491/ using document root <none> and password file <none>
I20260926 05:50:23.076527 28718 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:23.076603 28718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:23.078234 28718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:50:23.079071 28767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:23.079370 28718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:50:23.079478 28718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "431c41bef5494a289bfb4631d9ad325e"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "0fb5583e07fe3d3c4146cb509e8eb7f5"
server_key_iv: "0bffac5515d93c2ddedd36d99a2fc72e"
server_key_version: "encryptionkey@0"
I20260926 05:50:23.080045 28718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:23.119380 28718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:23.122550 28718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:23.122813 28718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:23.123174 28718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:23.123806 28774 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:50:23.125682 28718 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:50:23.125754 28718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:50:23.125794 28718 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:50:23.126911 28718 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:50:23.126976 28718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:23.127094 28774 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Bootstrap starting.
I20260926 05:50:23.135572 28718 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:41155
I20260926 05:50:23.135639 28881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:41155 every 8 connection(s)
I20260926 05:50:23.136234 28718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:23.140182 28774 log.cc:824] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:23.140591 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28718
I20260926 05:50:23.140934 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:46541
--local_ip_for_outbound_sockets=127.11.247.2
--tserver_master_addrs=127.11.247.62:40353
--webserver_port=37683
--webserver_interface=127.11.247.2
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:50:23.143044 28774 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:50:23.143383 28774 tablet_bootstrap.cc:492] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Bootstrap complete.
I20260926 05:50:23.144608 28774 ts_tablet_manager.cc:1397] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.014s
I20260926 05:50:23.145802 28774 raft_consensus.cc:348] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:23.146087 28774 raft_consensus.cc:729] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431c41bef5494a289bfb4631d9ad325e, State: Initialized, Role: FOLLOWER
I20260926 05:50:23.146302 28774 consensus_queue.cc:260] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:23.146674 28774 ts_tablet_manager.cc:1428] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260926 05:50:23.212774 28884 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:50:23.137198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:43131)
0926 05:50:23.137513 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.137572 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:23.138319 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:23.212454 (+ 74135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":177,"threads_started":1}
W20260926 05:50:23.213137 28882 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:40353 (0 consecutive failures): Network error: Failed to ping master at 127.11.247.62:40353: Client connection negotiation failed: client connection to 127.11.247.62:40353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:50:23.213661 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.213474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:41051)
0926 05:50:23.213553 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.213618 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 05:50:23.214130 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.213998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:34165)
0926 05:50:23.214082 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.214107 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:50:23.214251 28882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:50:23.251031 28886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:23.251319 28886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:23.251360 28886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:23.252602 28886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:23.252671 28886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:23.252717 28886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:23.254317 28886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:46541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=37683
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:23.254644 28886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:23.254913 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:23.260644 28895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:23.260673 28892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:23.260644 28893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:23.261036 28886 server_base.cc:1034] running on GCE node
I20260926 05:50:23.261232 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:23.261458 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:23.262661 28886 hybrid_clock.cc:648] HybridClock initialized: now 1790401823262642 us; error 28 us; skew 500 ppm
Sep 26 05:50:23 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401823, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:23.265589 28886 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:50:23.266189 28886 webserver.cc:460] Webserver started at http://127.11.247.2:37683/ using document root <none> and password file <none>
I20260926 05:50:23.266477 28886 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:23.266556 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:23.268230 28886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:23.269143 28902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:23.269407 28886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:23.269520 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "c12dab0c8692449291125efab5719dc2"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "9bb48636f016638d6b93059709feedc4"
server_key_iv: "4d4b972ed160931372a659c0cd84dbd9"
server_key_version: "encryptionkey@0"
I20260926 05:50:23.270028 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:23.285228 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:23.288028 28886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:23.288255 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:23.288585 28886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:23.289153 28909 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 05:50:23.290877 28909 ts_tablet_manager.cc:1852] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 05:50:23.291226 28909 ts_tablet_manager.cc:1905] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:23.293763 28909 ts_tablet_manager.cc:1918] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:23.293889 28909 log.cc:1196] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:23.294250 28909 ts_tablet_manager.cc:1939] T f8d9610f584c4fb58b6528fbee4a845d P c12dab0c8692449291125efab5719dc2: Deleting consensus metadata
I20260926 05:50:23.294508 28886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 05:50:23.294592 28886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 05:50:23.294690 28886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:23.294724 28886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:23.302198 28886 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:46541
I20260926 05:50:23.302271 29016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:46541 every 8 connection(s)
I20260926 05:50:23.302688 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:50:23.304008 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.303564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:39913)
0926 05:50:23.303741 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.303828 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
W20260926 05:50:23.304307 29017 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:40353 (0 consecutive failures): Network error: Failed to ping master at 127.11.247.62:40353: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
I20260926 05:50:23.304634 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.304533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:43809)
0926 05:50:23.304578 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.304614 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 05:50:23.304965 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.304894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:55955)
0926 05:50:23.304929 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.304951 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:50:23.305099 29017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:50:23.309449 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 28886
I20260926 05:50:23.309787 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:46251
--local_ip_for_outbound_sockets=127.11.247.3
--tserver_master_addrs=127.11.247.62:40353
--webserver_port=39805
--webserver_interface=127.11.247.3
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:23.401293 29020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:23.401520 29020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:23.401546 29020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:23.402853 29020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:23.402904 29020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:23.402933 29020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:23.404497 29020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:46251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=39805
--tserver_master_addrs=127.11.247.62:40353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:23.404806 29020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:23.405082 29020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:23.410291 29025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:23.410300 29026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:23.410370 29028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:23.410683 29020 server_base.cc:1034] running on GCE node
I20260926 05:50:23.410939 29020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:23.411221 29020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:23.412402 29020 hybrid_clock.cc:648] HybridClock initialized: now 1790401823412406 us; error 47 us; skew 500 ppm
Sep 26 05:50:23 dist-test-slave-21p4 krb5kdc[27578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401823, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:23.415210 29020 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:50:23.415783 29020 webserver.cc:460] Webserver started at http://127.11.247.3:39805/ using document root <none> and password file <none>
I20260926 05:50:23.416056 29020 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:23.416129 29020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:23.417610 29020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:23.418306 29035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:23.418501 29020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:23.418637 29020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "5f01fe5be374409db59eea82d308e170"
format_stamp: "Formatted at 2026-09-26 05:49:41 on dist-test-slave-21p4"
server_key: "63b26487ed3991c5c08280e4340a90bd"
server_key_iv: "54ebe9dc61e0a4a13e6bbe2a9f9c9a19"
server_key_version: "encryptionkey@0"
I20260926 05:50:23.419026 29020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:23.432093 29020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:23.434130 29020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:23.434327 29020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:23.434633 29020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:23.434989 29020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:23.435046 29020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:23.435087 29020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:23.435115 29020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:23.441545 29020 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:46251
I20260926 05:50:23.441584 29148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:46251 every 8 connection(s)
I20260926 05:50:23.441960 29020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:50:23.443109 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.442691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:42041)
0926 05:50:23.442846 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.442955 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
W20260926 05:50:23.443512 29149 heartbeater.cc:644] Failed to heartbeat to 127.11.247.62:40353 (0 consecutive failures): Network error: Failed to ping master at 127.11.247.62:40353: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
I20260926 05:50:23.443820 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.443714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:57511)
0926 05:50:23.443768 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.443800 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:50:23.444104 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:23.444012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:50007)
0926 05:50:23.444056 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:23.444078 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:50:23.444201 29149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:50:23.446116 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29020
I20260926 05:50:24.214932 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.214676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:37819)
0926 05:50:24.214833 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:24.214871 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260926 05:50:24.305748 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.305513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:42849)
0926 05:50:24.305658 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:24.305705 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260926 05:50:24.444808 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.444576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:49279)
0926 05:50:24.444706 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:24.444750 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260926 05:50:24.666813 29156 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:24.666961 29156 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:24.667433 29156 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
Sep 26 05:50:24 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401823, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.3@KRBTEST.COM
Sep 26 05:50:24 dist-test-slave-21p4 krb5kdc[27578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401823, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.2@KRBTEST.COM
I20260926 05:50:24.679188 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.667748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:41495 (local address 127.11.247.3:46251)
0926 05:50:24.667926 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:24.667936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:24.667975 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:50:24.668504 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:24.668506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:24.668549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:24.668701 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:24.669509 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.670395 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.671601 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.672242 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.672380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:24.675117 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:24.675123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:24.675123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:24.675138 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:24.677100 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:24.677636 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:24.677639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:24.677639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:24.677774 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:24.678079 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:24.678080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:24.678080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:24.678431 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:24.678518 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:24.678921 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 05:50:24.679064 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":49,"threads_started":1}
I20260926 05:50:24.679188 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.667709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:44915 (local address 127.11.247.2:46541)
0926 05:50:24.667915 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:24.667928 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:24.667960 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:50:24.668500 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:24.668503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:24.668549 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:24.668701 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:24.669493 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.670380 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.671630 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.672241 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.672360 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:24.675486 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:24.675490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:24.675490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:24.675502 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:24.677338 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:24.677730 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:24.677732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:24.677732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:24.677878 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:24.678079 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:24.678080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:24.678080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:24.678431 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:24.678518 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:24.678933 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 05:50:24.679064 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260926 05:50:24.679185 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.667611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:46251 (local address 127.11.247.1:41495)
0926 05:50:24.667751 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:24.667765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:24.667834 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:24.668757 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:24.668760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:24.669134 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:24.669360 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:24.669366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.670521 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.670523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:24.671521 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:24.671524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.672350 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.672351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:24.672452 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:24.673125 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:24.673135 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:24.674985 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:24.677196 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:24.677204 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:24.677206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:24.677551 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:24.677847 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:24.677847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:24.677848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:24.677994 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:24.678529 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:24.678536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:24.678721 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 05:50:24.678845 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 05:50:24.679066 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":119}
I20260926 05:50:24.679235 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:24.667630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:46541 (local address 127.11.247.1:44915)
0926 05:50:24.667764 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:24.667777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:24.667834 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:24.668756 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:24.668759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:24.669134 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:24.669360 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:24.669366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.670498 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.670499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:24.671518 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:24.671521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:24.672476 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:24.672477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:24.672530 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:24.673176 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:24.673180 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:24.675374 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:24.677417 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:24.677417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:24.677418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:24.677649 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:24.677953 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:24.677953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:24.677953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:24.678007 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:24.678527 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:24.678536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:24.678721 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 05:50:24.678874 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 05:50:24.679121 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":223,"thread_start_us":63,"threads_started":1}
W20260926 05:50:24.680172 28771 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
W20260926 05:50:24.680918 28769 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:24.680982 28769 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e; no voters: 5f01fe5be374409db59eea82d308e170, c12dab0c8692449291125efab5719dc2
I20260926 05:50:24.681088 29156 raft_consensus.cc:2726] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:50:25.215682 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:25.215448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:33849)
0926 05:50:25.215592 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:25.215632 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260926 05:50:25.306538 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:25.306308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:41501)
0926 05:50:25.306449 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:25.306492 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260926 05:50:25.445611 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:25.445356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:50739)
0926 05:50:25.445511 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:25.445559 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260926 05:50:26.216452 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:26.216219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:42953)
0926 05:50:26.216358 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:26.216397 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260926 05:50:26.307269 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:26.307057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:35883)
0926 05:50:26.307181 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:26.307222 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 05:50:26.446408 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:26.446125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:42539)
0926 05:50:26.446259 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:26.446302 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260926 05:50:26.851344 29166 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:26.851446 29166 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:26.851676 29166 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
W20260926 05:50:26.852326 28769 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
W20260926 05:50:26.852401 28771 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:26.852430 28769 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e; no voters: 5f01fe5be374409db59eea82d308e170, c12dab0c8692449291125efab5719dc2
I20260926 05:50:26.852517 29166 raft_consensus.cc:2726] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:50:27.217317 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:27.217075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:44203)
0926 05:50:27.217220 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:27.217265 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260926 05:50:27.307963 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:27.307727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:56647)
0926 05:50:27.307875 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:27.307916 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260926 05:50:27.447165 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:27.446938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:60987)
0926 05:50:27.447074 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:27.447115 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260926 05:50:28.218109 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:28.217873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:48023)
0926 05:50:28.218023 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:28.218062 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260926 05:50:28.308779 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:28.308530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:40747)
0926 05:50:28.308681 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:28.308727 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260926 05:50:28.447954 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:28.447701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:55215)
0926 05:50:28.447841 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:28.447887 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260926 05:50:29.218883 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:29.218659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:46245)
0926 05:50:29.218792 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:29.218835 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260926 05:50:29.309614 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:29.309386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:43149)
0926 05:50:29.309529 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:29.309568 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260926 05:50:29.448771 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:29.448537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:40521)
0926 05:50:29.448684 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:29.448725 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260926 05:50:29.920840 29176 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:29.920946 29176 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:29.921176 29176 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
W20260926 05:50:29.921646 28771 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
W20260926 05:50:29.921754 28769 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:29.921813 28769 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e; no voters: 5f01fe5be374409db59eea82d308e170, c12dab0c8692449291125efab5719dc2
I20260926 05:50:29.921885 29176 raft_consensus.cc:2726] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:50:30.219705 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:30.219441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:52935)
0926 05:50:30.219608 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:30.219654 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260926 05:50:30.310444 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:30.310186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:43389)
0926 05:50:30.310336 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:30.310381 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260926 05:50:30.449671 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:30.449404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:37919)
0926 05:50:30.449579 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:30.449620 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260926 05:50:31.220536 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:31.220308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:48141)
0926 05:50:31.220441 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:31.220489 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260926 05:50:31.311411 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:31.311168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:44777)
0926 05:50:31.311315 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:31.311362 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260926 05:50:31.450481 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:31.450221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:40679)
0926 05:50:31.450390 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:31.450429 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260926 05:50:32.221418 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:32.221123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:39867)
0926 05:50:32.221285 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:32.221334 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260926 05:50:32.264492 29184 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:32.264590 29184 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:32.264792 29184 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
W20260926 05:50:32.265208 28769 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
W20260926 05:50:32.265297 28771 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:32.265396 28771 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e; no voters: 5f01fe5be374409db59eea82d308e170, c12dab0c8692449291125efab5719dc2
I20260926 05:50:32.265515 29184 raft_consensus.cc:2726] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:50:32.312260 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:32.311990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:40781)
0926 05:50:32.312164 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:32.312210 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260926 05:50:32.451246 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:32.450964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:58487)
0926 05:50:32.451146 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:32.451191 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260926 05:50:33.222328 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:33.222063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:42885)
0926 05:50:33.222220 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:33.222276 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260926 05:50:33.313138 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:33.312852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:39437)
0926 05:50:33.313038 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:33.313088 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260926 05:50:33.452023 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:33.451758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:46283)
0926 05:50:33.451918 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:33.451964 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260926 05:50:34.197132 29190 raft_consensus.cc:482] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:34.197234 29190 raft_consensus.cc:504] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12dab0c8692449291125efab5719dc2" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 46541 } } peers { permanent_uuid: "5f01fe5be374409db59eea82d308e170" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46251 } } peers { permanent_uuid: "431c41bef5494a289bfb4631d9ad325e" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41155 } }
I20260926 05:50:34.197440 29190 leader_election.cc:288] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541), 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251)
W20260926 05:50:34.197847 28771 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c12dab0c8692449291125efab5719dc2 (127.11.247.2:46541): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
W20260926 05:50:34.197947 28769 leader_election.cc:341] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f01fe5be374409db59eea82d308e170 (127.11.247.3:46251): Not found: Tablet not found: f8d9610f584c4fb58b6528fbee4a845d
I20260926 05:50:34.198024 28769 leader_election.cc:302] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 431c41bef5494a289bfb4631d9ad325e; no voters: 5f01fe5be374409db59eea82d308e170, c12dab0c8692449291125efab5719dc2
I20260926 05:50:34.198107 29190 raft_consensus.cc:2726] T f8d9610f584c4fb58b6528fbee4a845d P 431c41bef5494a289bfb4631d9ad325e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:50:34.223237 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:34.222968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:34945)
0926 05:50:34.223149 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:34.223188 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260926 05:50:34.313973 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:34.313717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:57617)
0926 05:50:34.313880 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:34.313925 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260926 05:50:34.452775 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:34.452508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:52521)
0926 05:50:34.452656 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:34.452709 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260926 05:50:35.224045 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:35.223800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:53303)
0926 05:50:35.223950 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:35.223994 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260926 05:50:35.314778 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:35.314512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:33031)
0926 05:50:35.314686 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:35.314724 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260926 05:50:35.453593 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:35.453322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:35959)
0926 05:50:35.453491 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:35.453540 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260926 05:50:36.224777 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:36.224531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:33673)
0926 05:50:36.224683 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:36.224726 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260926 05:50:36.315606 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:36.315332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:43317)
0926 05:50:36.315494 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:36.315556 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260926 05:50:36.454334 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:36.454086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:45005)
0926 05:50:36.454237 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:36.454281 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260926 05:50:37.225776 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:37.225512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:43257)
0926 05:50:37.225681 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:37.225727 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260926 05:50:37.316445 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:37.316160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:38435)
0926 05:50:37.316346 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:37.316394 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260926 05:50:37.455083 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:37.454870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:47665)
0926 05:50:37.454992 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:37.455032 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260926 05:50:38.226670 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:38.226393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.1:55723)
0926 05:50:38.226520 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:38.226563 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 05:50:38.317298 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:38.317007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.2:60957)
0926 05:50:38.317175 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:38.317244 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260926 05:50:38.453240 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 28718
I20260926 05:50:38.455917 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:38.455660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:40353 (local address 127.11.247.3:39941)
0926 05:50:38.455834 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:38.455872 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.62:40353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
I20260926 05:50:38.458793 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 28886
I20260926 05:50:38.463301 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29020
2026-09-26T05:50:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58537 ms)
[----------] 30 tests from SecurityITest (243221 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:38 dist-test-slave-21p4 krb5kdc[29208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:38 dist-test-slave-21p4 krb5kdc[29208](info): set up 2 sockets
Sep 26 05:50:38 dist-test-slave-21p4 krb5kdc[29208](info): commencing operation
krb5kdc: starting...
W20260926 05:50:39.502710 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:39 dist-test-slave-21p4 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:50:39.630465 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39421
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:39.714267 29221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:39.714485 29221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:39.714511 29221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:39.715804 29221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:39.715853 29221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:39.715875 29221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:39.715888 29221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:39.715901 29221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:39.717382 29221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:39421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:39.717598 29221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:39.717783 29221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:39.722649 29226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:39.722703 29227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:39.722929 29221 server_base.cc:1034] running on GCE node
W20260926 05:50:39.722703 29229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:39.723258 29221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:39.723559 29221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:39.724743 29221 hybrid_clock.cc:648] HybridClock initialized: now 1790401839724715 us; error 72 us; skew 500 ppm
Sep 26 05:50:39 dist-test-slave-21p4 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401839, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:39.727473 29221 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:50:39.728006 29221 webserver.cc:460] Webserver started at http://127.11.247.62:43597/ using document root <none> and password file <none>
I20260926 05:50:39.728250 29221 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:39.728317 29221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:39.728482 29221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:39.729579 29221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "fd5042cc5b734e85961bdbef56b9e202"
format_stamp: "Formatted at 2026-09-26 05:50:39 on dist-test-slave-21p4"
server_key: "f8115754bb08036310b5f239a286c127"
server_key_iv: "4b8f58f2141457055998a58a617d8186"
server_key_version: "encryptionkey@0"
I20260926 05:50:39.729959 29221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "fd5042cc5b734e85961bdbef56b9e202"
format_stamp: "Formatted at 2026-09-26 05:50:39 on dist-test-slave-21p4"
server_key: "f8115754bb08036310b5f239a286c127"
server_key_iv: "4b8f58f2141457055998a58a617d8186"
server_key_version: "encryptionkey@0"
I20260926 05:50:39.731474 29221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260926 05:50:39.732462 29236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:39.732661 29221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:39.732776 29221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "fd5042cc5b734e85961bdbef56b9e202"
format_stamp: "Formatted at 2026-09-26 05:50:39 on dist-test-slave-21p4"
server_key: "f8115754bb08036310b5f239a286c127"
server_key_iv: "4b8f58f2141457055998a58a617d8186"
server_key_version: "encryptionkey@0"
I20260926 05:50:39.732877 29221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:39.753415 29221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:39.756014 29221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:39.756197 29221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:39.760633 29221 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:39421
I20260926 05:50:39.760684 29288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:39421 every 8 connection(s)
I20260926 05:50:39.761085 29221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:39.762036 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:39.764592 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202: Bootstrap starting.
I20260926 05:50:39.765194 29289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:39.765455 29289 log.cc:824] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:39.765980 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29221
I20260926 05:50:39.766101 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:39.766356 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202: No bootstrap required, opened a new log
I20260926 05:50:39.766366 12252 external_mini_cluster.cc:1428] Setting key d23b7d7e912229493a9fd81388aceb0d
I20260926 05:50:39.768982 29289 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5042cc5b734e85961bdbef56b9e202" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39421 } }
I20260926 05:50:39.769133 29289 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:39.769156 29289 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5042cc5b734e85961bdbef56b9e202, State: Initialized, Role: FOLLOWER
I20260926 05:50:39.769294 29289 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5042cc5b734e85961bdbef56b9e202" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39421 } }
I20260926 05:50:39.769371 29289 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:39.769424 29289 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:39.769474 29289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:39.770097 29289 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5042cc5b734e85961bdbef56b9e202" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39421 } }
I20260926 05:50:39.770186 29289 leader_election.cc:302] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd5042cc5b734e85961bdbef56b9e202; no voters:
I20260926 05:50:39.770375 29289 leader_election.cc:288] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:39.770398 29294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:39.770514 29294 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [term 1 LEADER]: Becoming Leader. State: Replica: fd5042cc5b734e85961bdbef56b9e202, State: Running, Role: LEADER
I20260926 05:50:39.770689 29289 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:39.770684 29294 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5042cc5b734e85961bdbef56b9e202" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39421 } }
I20260926 05:50:39.771209 29296 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd5042cc5b734e85961bdbef56b9e202. Latest consensus state: current_term: 1 leader_uuid: "fd5042cc5b734e85961bdbef56b9e202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5042cc5b734e85961bdbef56b9e202" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39421 } } }
I20260926 05:50:39.771302 29296 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:39.771552 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd5042cc5b734e85961bdbef56b9e202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd5042cc5b734e85961bdbef56b9e202" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39421 } } }
I20260926 05:50:39.771898 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:39.772065 29302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:39.772697 29302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:50:39 dist-test-slave-21p4 krb5kdc[29208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:39.775056 29302 catalog_manager.cc:1347] Generated new cluster ID: 20218ab9ddd9460eb7729a94fcf59073
I20260926 05:50:39.775120 29302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:39.779559 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:39.766997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.11.247.62:39421)
0926 05:50:39.767214 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:39.767228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:39.767257 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:50:39.767877 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:39.767885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:39.767949 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:39.768073 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:39.768935 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:39.770365 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:39.771521 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:39.772182 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:39.772291 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:39.775390 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:39.775394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:39.775394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:39.775407 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:39.777579 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:39.778021 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:39.778024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:39.778025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:39.778141 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:39.778393 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:39.778394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:39.778395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:39.778811 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:39.779018 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:39.779106 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 05:50:39.779297 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260926 05:50:39.793267 29302 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:39.794054 29302 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:39.804318 29302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202: Generated new TSK 0
I20260926 05:50:39.804561 29302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:50:39.838989 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39421
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:39.927664 29315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:39.927884 29315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:39.927910 29315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:39.929083 29315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:39.929123 29315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:39.929150 29315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:39.930653 29315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:39.930866 29315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:39.931059 29315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:39.936309 29320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:39.936313 29323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:39.936483 29315 server_base.cc:1034] running on GCE node
W20260926 05:50:39.936309 29321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:39.936741 29315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:39.937032 29315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:39.938205 29315 hybrid_clock.cc:648] HybridClock initialized: now 1790401839938192 us; error 33 us; skew 500 ppm
Sep 26 05:50:39 dist-test-slave-21p4 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401839, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:39.941056 29315 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:50:39.941560 29315 webserver.cc:460] Webserver started at http://127.11.247.1:46635/ using document root <none> and password file <none>
I20260926 05:50:39.941814 29315 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:39.941886 29315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:39.942039 29315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:39.943215 29315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "027a66dda1f7407c9d4f64f01dbfd63b"
format_stamp: "Formatted at 2026-09-26 05:50:39 on dist-test-slave-21p4"
server_key: "fbbc4b3e9bf493abdb82ce534e06668b"
server_key_iv: "8ad6956746628d65be38434c0fd9a2c0"
server_key_version: "encryptionkey@0"
I20260926 05:50:39.943607 29315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "027a66dda1f7407c9d4f64f01dbfd63b"
format_stamp: "Formatted at 2026-09-26 05:50:39 on dist-test-slave-21p4"
server_key: "fbbc4b3e9bf493abdb82ce534e06668b"
server_key_iv: "8ad6956746628d65be38434c0fd9a2c0"
server_key_version: "encryptionkey@0"
I20260926 05:50:39.945127 29315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:39.946048 29330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:39.946345 29315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:39.946494 29315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "027a66dda1f7407c9d4f64f01dbfd63b"
format_stamp: "Formatted at 2026-09-26 05:50:39 on dist-test-slave-21p4"
server_key: "fbbc4b3e9bf493abdb82ce534e06668b"
server_key_iv: "8ad6956746628d65be38434c0fd9a2c0"
server_key_version: "encryptionkey@0"
I20260926 05:50:39.946588 29315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:39.966511 29315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:39.969125 29315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:39.969336 29315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:39.969609 29315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:39.969991 29315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:39.970055 29315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:39.970093 29315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:39.970140 29315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:39.976598 29315 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:33755
I20260926 05:50:39.976647 29443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:33755 every 8 connection(s)
I20260926 05:50:39.977008 29315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:50:39 dist-test-slave-21p4 krb5kdc[29208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401839, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:39.984586 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29315
I20260926 05:50:39.984756 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:50:39.985026 12252 external_mini_cluster.cc:1428] Setting key d1966114b1deb981f1a8e479642c4ca1
I20260926 05:50:39.988528 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:39.977834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:43511 (local address 127.11.247.62:39421)
0926 05:50:39.977927 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:39.977930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:39.978132 (+ 202us) server_negotiation.cc:407] Connection header received
0926 05:50:39.978712 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:39.978714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:39.978727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:39.978784 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:39.979803 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:39.980335 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:39.981420 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:39.981648 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:39.981731 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:39.984241 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:39.984243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:39.984243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:39.984251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:39.986357 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:39.986919 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:39.986921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:39.986921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:39.986992 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:39.987450 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:39.987451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:39.987452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:39.987594 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:39.987686 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:39.988272 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 05:50:39.988415 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:50:39.988734 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:39.977797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39421 (local address 127.11.247.1:43511)
0926 05:50:39.977999 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:39.978039 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:39.978571 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:39.978866 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:39.978869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:39.979475 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:39.979692 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:39.979697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:39.980460 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:39.980466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:39.981309 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:39.981312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:39.981742 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:39.981743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:39.981882 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:39.982466 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:39.982479 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:39.984116 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:39.986478 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:39.986488 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:39.986490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:39.986839 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:39.987095 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:39.987096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:39.987096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:39.987313 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:39.987675 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:39.987684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:39.987847 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 05:50:39.987970 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 05:50:39.988450 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260926 05:50:39.989316 29444 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39421
I20260926 05:50:39.989439 29444 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:39.989657 29444 heartbeater.cc:511] Master 127.11.247.62:39421 requested a full tablet report, sending...
I20260926 05:50:39.990296 29253 ts_manager.cc:194] Registered new tserver with Master: 027a66dda1f7407c9d4f64f01dbfd63b (127.11.247.1:33755)
I20260926 05:50:39.991097 29253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:43511
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:50:40.019295 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39421
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:40.106356 29449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:40.106650 29449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:40.106679 29449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:40.108075 29449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:40.108152 29449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:40.108191 29449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:40.109972 29449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:40.110283 29449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:40.110608 29449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:40.116178 29455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.116178 29457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.116442 29454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:40.116479 29449 server_base.cc:1034] running on GCE node
I20260926 05:50:40.116734 29449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:40.117046 29449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:40.118310 29449 hybrid_clock.cc:648] HybridClock initialized: now 1790401840118305 us; error 43 us; skew 500 ppm
Sep 26 05:50:40 dist-test-slave-21p4 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401840, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:40.121169 29449 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:50:40.121628 29449 webserver.cc:460] Webserver started at http://127.11.247.2:44149/ using document root <none> and password file <none>
I20260926 05:50:40.121835 29449 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:40.121879 29449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:40.121986 29449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:40.123164 29449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "87e5dd7b39444bdeb6667d3309e0861d"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "8d3396c823997c8d16fce842dbddce38"
server_key_iv: "6c6cbad70ea51a4e875c62a3098e2772"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.123495 29449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "87e5dd7b39444bdeb6667d3309e0861d"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "8d3396c823997c8d16fce842dbddce38"
server_key_iv: "6c6cbad70ea51a4e875c62a3098e2772"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.124936 29449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:40.125864 29464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.126086 29449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:40.126178 29449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "87e5dd7b39444bdeb6667d3309e0861d"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "8d3396c823997c8d16fce842dbddce38"
server_key_iv: "6c6cbad70ea51a4e875c62a3098e2772"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.126245 29449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:40.137959 29449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:40.140241 29449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:40.140372 29449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:40.140584 29449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:40.141111 29449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:40.141160 29449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.141185 29449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:40.141252 29449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.147310 29449 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35847
I20260926 05:50:40.147425 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35847 every 8 connection(s)
I20260926 05:50:40.147742 29449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:50:40 dist-test-slave-21p4 krb5kdc[29208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401840, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:40.155189 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29449
I20260926 05:50:40.155299 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:50:40.155535 12252 external_mini_cluster.cc:1428] Setting key a719bce209b356a73cd6c268f1f7e412
I20260926 05:50:40.158665 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.148537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:57113 (local address 127.11.247.62:39421)
0926 05:50:40.148614 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.148616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.148849 (+ 233us) server_negotiation.cc:407] Connection header received
0926 05:50:40.149441 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.149442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.149453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.149508 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:40.150677 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.151189 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.152215 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.152414 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.152473 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.154906 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:40.154908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:40.154908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:40.154916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:40.156699 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:40.157297 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:40.157298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:40.157299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:40.157366 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:40.157795 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:40.157796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:40.157797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:40.158010 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:40.158111 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.158394 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.158501 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:50:40.158771 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.148538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39421 (local address 127.11.247.2:57113)
0926 05:50:40.148719 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.148758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.149320 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.149577 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.149580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.150335 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:40.150553 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.150558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.151279 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.151279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.152126 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.152128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.152526 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.152527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.152657 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.153262 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:40.153271 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:40.154811 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:40.156798 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:40.156807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:40.156810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:40.157161 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:40.157454 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:40.157455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:40.157455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:40.157643 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:40.158104 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:40.158110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:40.158246 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.158338 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.158491 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260926 05:50:40.159590 29578 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39421
I20260926 05:50:40.159745 29578 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:40.159994 29578 heartbeater.cc:511] Master 127.11.247.62:39421 requested a full tablet report, sending...
I20260926 05:50:40.160468 29253 ts_manager.cc:194] Registered new tserver with Master: 87e5dd7b39444bdeb6667d3309e0861d (127.11.247.2:35847)
I20260926 05:50:40.160897 29253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:57113
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:50:40.186239 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39421
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:40.269598 29583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:40.269847 29583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:40.269891 29583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:40.271255 29583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:40.271317 29583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:40.271365 29583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:40.273051 29583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:40.273346 29583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:40.273592 29583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:40.278836 29588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.278841 29591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.278980 29589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:40.279080 29583 server_base.cc:1034] running on GCE node
I20260926 05:50:40.279320 29583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:40.279654 29583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:40.280817 29583 hybrid_clock.cc:648] HybridClock initialized: now 1790401840280824 us; error 49 us; skew 500 ppm
Sep 26 05:50:40 dist-test-slave-21p4 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401840, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:40.283552 29583 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:50:40.284122 29583 webserver.cc:460] Webserver started at http://127.11.247.3:33705/ using document root <none> and password file <none>
I20260926 05:50:40.284387 29583 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:40.284467 29583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:40.284615 29583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:40.285789 29583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "44cd59eb508e40f3af451ae1ef705aa4"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "a7ce391b74b3682007ed8d4da6f46dfa"
server_key_iv: "c312942c65ff4734cc1389b61d082610"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.286170 29583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "44cd59eb508e40f3af451ae1ef705aa4"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "a7ce391b74b3682007ed8d4da6f46dfa"
server_key_iv: "c312942c65ff4734cc1389b61d082610"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.287712 29583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:50:40.288544 29598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.288772 29583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:40.288880 29583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "44cd59eb508e40f3af451ae1ef705aa4"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "a7ce391b74b3682007ed8d4da6f46dfa"
server_key_iv: "c312942c65ff4734cc1389b61d082610"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.288964 29583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:40.303256 29583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:40.305840 29583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:40.306015 29583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:40.306267 29583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:40.306689 29583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:40.306744 29583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.306790 29583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:40.306816 29583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.312922 29583 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:46813
I20260926 05:50:40.313001 29711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:46813 every 8 connection(s)
I20260926 05:50:40.313326 29583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:50:40 dist-test-slave-21p4 krb5kdc[29208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401840, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:40.321764 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29583
I20260926 05:50:40.321874 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:50:40.322108 12252 external_mini_cluster.cc:1428] Setting key 8de413315e99420a2dc7a7678cde47d0
I20260926 05:50:40.322876 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.314153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:59733 (local address 127.11.247.62:39421)
0926 05:50:40.314224 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.314226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.314418 (+ 192us) server_negotiation.cc:407] Connection header received
0926 05:50:40.314959 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.314961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.314971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.315033 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:40.316042 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.316535 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.317513 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.317699 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.317793 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.319846 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:40.319848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:40.319848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:40.319854 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:40.321268 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:40.321655 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:40.321657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:40.321657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:40.321728 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:40.322088 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:40.322090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:40.322090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:40.322237 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:40.322316 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.322676 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.322782 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:50:40.323079 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.314128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:39421 (local address 127.11.247.3:59733)
0926 05:50:40.314292 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.314334 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.314875 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.315086 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.315089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.315764 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:40.315961 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.315966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.316623 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.316623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.317426 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.317428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.317776 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.317777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.317879 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.318336 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:40.318344 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:40.319768 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:40.321349 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:40.321355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:40.321357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:40.321581 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:40.321891 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:40.321892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:40.321892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:40.322020 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:40.322302 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:40.322309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:40.322517 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.322653 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.322833 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260926 05:50:40.323580 29712 heartbeater.cc:347] Connected to a master server at 127.11.247.62:39421
I20260926 05:50:40.323688 29712 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:40.323866 29712 heartbeater.cc:511] Master 127.11.247.62:39421 requested a full tablet report, sending...
I20260926 05:50:40.324227 29253 ts_manager.cc:194] Registered new tserver with Master: 44cd59eb508e40f3af451ae1ef705aa4 (127.11.247.3:46813)
I20260926 05:50:40.324612 29253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:59733
I20260926 05:50:40.334028 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:40.341979 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.335618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.11.247.62:39421)
0926 05:50:40.335672 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.335675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.335766 (+ 91us) server_negotiation.cc:407] Connection header received
0926 05:50:40.335829 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.335830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.335839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.335928 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:40.336512 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.337009 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.337745 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.337937 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.338013 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.339295 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:40.339297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:40.339297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:40.339304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:40.340768 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:40.341142 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:40.341143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:40.341144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:40.341206 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:40.341410 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:40.341412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:40.341412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:40.341573 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:40.341681 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.341793 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.341873 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:50:40.343727 29253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:40.345310 29253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:40.352710 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.348423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.11.247.3:46813)
0926 05:50:40.348616 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.348623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.348635 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:50:40.348677 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.348680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.348697 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.349138 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:40.349211 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.350318 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.351501 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.352275 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.352357 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.352427 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.352582 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.352626 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260926 05:50:40.352739 29724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.348008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:46813 (local address 127.0.0.1:39452)
0926 05:50:40.348228 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.348271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.348335 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.348829 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.348832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.348837 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:40.348942 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.348945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.350434 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.350435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.351408 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.351410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.352361 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.352362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.352458 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.352476 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.352556 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.352636 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":40,"thread_start_us":109,"threads_started":1}
I20260926 05:50:40.353077 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.348179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.11.247.1:33755)
0926 05:50:40.348670 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.348673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.348682 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:40.348718 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.348719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.348736 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.348909 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:40.349095 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.350275 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.351632 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.352354 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.352415 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.352461 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.352960 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.353005 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":121,"threads_started":1}
I20260926 05:50:40.352761 29723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.348020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33755 (local address 127.0.0.1:39988)
0926 05:50:40.348228 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.348270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.348335 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.348831 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.348832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.348834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:40.349021 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.349022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.350386 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.350388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.351539 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.351542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.352430 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.352431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.352503 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.352506 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.352557 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.352636 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":170,"thread_start_us":86,"threads_started":1}
I20260926 05:50:40.353657 29646 tablet_service.cc:1467] Processing CreateTablet for tablet 08f58147f7e04910ab93841282c08ec5 (DEFAULT_TABLE table=test-table [id=bc35a3407ff8439f8d40c12c54813cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:40.353906 29378 tablet_service.cc:1467] Processing CreateTablet for tablet 08f58147f7e04910ab93841282c08ec5 (DEFAULT_TABLE table=test-table [id=bc35a3407ff8439f8d40c12c54813cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:40.354007 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f58147f7e04910ab93841282c08ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:40.354228 29378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f58147f7e04910ab93841282c08ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:40.356855 29730 tablet_bootstrap.cc:492] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Bootstrap starting.
I20260926 05:50:40.356899 29729 tablet_bootstrap.cc:492] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: Bootstrap starting.
I20260926 05:50:40.357587 29730 tablet_bootstrap.cc:654] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:40.357591 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.348253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35847 (local address 127.0.0.1:38616)
0926 05:50:40.348681 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.348690 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.348752 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.349026 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.349027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.349029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:40.349081 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.349083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.350074 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.350075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.351540 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.351542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.357257 (+ 5715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.357259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.357349 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.357356 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.357428 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.357483 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":7,"thread_start_us":75,"threads_started":1}
I20260926 05:50:40.357606 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.348500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.11.247.2:35847)
0926 05:50:40.348660 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.348664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.348812 (+ 148us) server_negotiation.cc:407] Connection header received
0926 05:50:40.348857 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.348859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.348874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.348944 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:40.349214 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.349972 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.352498 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.357148 (+ 4650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.357250 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.357320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.357452 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.357511 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260926 05:50:40.357656 29729 tablet_bootstrap.cc:654] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:40.358193 29730 log.cc:824] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:40.358239 29729 log.cc:824] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:40.358793 29512 tablet_service.cc:1467] Processing CreateTablet for tablet 08f58147f7e04910ab93841282c08ec5 (DEFAULT_TABLE table=test-table [id=bc35a3407ff8439f8d40c12c54813cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:40.359143 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f58147f7e04910ab93841282c08ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:40.359230 29729 tablet_bootstrap.cc:492] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: No bootstrap required, opened a new log
I20260926 05:50:40.359292 29729 ts_tablet_manager.cc:1397] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:50:40.360881 29729 raft_consensus.cc:348] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.361039 29729 raft_consensus.cc:374] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:40.361063 29729 raft_consensus.cc:729] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027a66dda1f7407c9d4f64f01dbfd63b, State: Initialized, Role: FOLLOWER
I20260926 05:50:40.361196 29729 consensus_queue.cc:260] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.361501 29729 ts_tablet_manager.cc:1428] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:50:40.361536 29444 heartbeater.cc:503] Master 127.11.247.62:39421 was elected leader, sending a full tablet report...
I20260926 05:50:40.362221 29734 tablet_bootstrap.cc:492] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Bootstrap starting.
I20260926 05:50:40.362447 29730 tablet_bootstrap.cc:492] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: No bootstrap required, opened a new log
I20260926 05:50:40.362542 29730 ts_tablet_manager.cc:1397] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 05:50:40.362958 29734 tablet_bootstrap.cc:654] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:40.363564 29734 log.cc:824] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:40.364400 29734 tablet_bootstrap.cc:492] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: No bootstrap required, opened a new log
I20260926 05:50:40.364497 29734 ts_tablet_manager.cc:1397] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:50:40.364815 29730 raft_consensus.cc:348] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.364974 29730 raft_consensus.cc:374] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:40.364997 29730 raft_consensus.cc:729] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cd59eb508e40f3af451ae1ef705aa4, State: Initialized, Role: FOLLOWER
I20260926 05:50:40.365140 29730 consensus_queue.cc:260] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.365456 29730 ts_tablet_manager.cc:1428] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:50:40.365483 29712 heartbeater.cc:503] Master 127.11.247.62:39421 was elected leader, sending a full tablet report...
I20260926 05:50:40.367038 29734 raft_consensus.cc:348] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.367210 29734 raft_consensus.cc:374] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:40.367240 29734 raft_consensus.cc:729] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e5dd7b39444bdeb6667d3309e0861d, State: Initialized, Role: FOLLOWER
I20260926 05:50:40.367368 29734 consensus_queue.cc:260] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.367638 29734 ts_tablet_manager.cc:1428] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260926 05:50:40.367684 29578 heartbeater.cc:503] Master 127.11.247.62:39421 was elected leader, sending a full tablet report...
W20260926 05:50:40.398483 29579 tablet.cc:2369] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:50:40.477898 29445 tablet.cc:2369] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:50:40.529999 29736 raft_consensus.cc:482] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:40.530206 29736 raft_consensus.cc:504] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.530722 29736 leader_election.cc:288] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 027a66dda1f7407c9d4f64f01dbfd63b (127.11.247.1:33755), 87e5dd7b39444bdeb6667d3309e0861d (127.11.247.2:35847)
I20260926 05:50:40.534981 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.531166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:36101 (local address 127.11.247.2:35847)
0926 05:50:40.531211 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.531213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.531347 (+ 134us) server_negotiation.cc:407] Connection header received
0926 05:50:40.531380 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.531381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.531391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.531456 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:40.531592 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.532296 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.533853 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.534539 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.534653 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.534665 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.534842 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.534897 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:50:40.535061 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.530934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33755 (local address 127.11.247.3:60963)
0926 05:50:40.530998 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.531008 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.531071 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.531251 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.531252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.531254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:40.531357 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.531360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.532289 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.532290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.533715 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.533718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.534695 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.534697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.534861 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.534865 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.534921 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.534982 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":82}
I20260926 05:50:40.535212 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.531064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35847 (local address 127.11.247.3:36101)
0926 05:50:40.531246 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.531280 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.531341 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.531461 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.531461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.531463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:40.531532 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.531533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.532394 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.532395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.533715 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.533718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.534654 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.534655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.534737 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.534743 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.535083 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.535134 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":5,"thread_start_us":114,"threads_started":1}
I20260926 05:50:40.535331 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.531030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:60963 (local address 127.11.247.1:33755)
0926 05:50:40.531097 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.531100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.531109 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:40.531137 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.531138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.531148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.531224 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:40.531442 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.532177 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.533819 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.534429 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.534515 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.534527 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.535223 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.535269 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:50:40.536507 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f58147f7e04910ab93841282c08ec5" candidate_uuid: "44cd59eb508e40f3af451ae1ef705aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e5dd7b39444bdeb6667d3309e0861d" is_pre_election: true
I20260926 05:50:40.536702 29532 raft_consensus.cc:2454] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44cd59eb508e40f3af451ae1ef705aa4 in term 0.
I20260926 05:50:40.536962 29600 leader_election.cc:302] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44cd59eb508e40f3af451ae1ef705aa4, 87e5dd7b39444bdeb6667d3309e0861d; no voters:
I20260926 05:50:40.536957 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f58147f7e04910ab93841282c08ec5" candidate_uuid: "44cd59eb508e40f3af451ae1ef705aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" is_pre_election: true
I20260926 05:50:40.537096 29398 raft_consensus.cc:2454] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44cd59eb508e40f3af451ae1ef705aa4 in term 0.
I20260926 05:50:40.537110 29736 raft_consensus.cc:2781] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:40.537158 29736 raft_consensus.cc:482] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:40.537191 29736 raft_consensus.cc:3037] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:40.538334 29736 raft_consensus.cc:504] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.538475 29736 leader_election.cc:288] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [CANDIDATE]: Term 1 election: Requested vote from peers 027a66dda1f7407c9d4f64f01dbfd63b (127.11.247.1:33755), 87e5dd7b39444bdeb6667d3309e0861d (127.11.247.2:35847)
I20260926 05:50:40.538658 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f58147f7e04910ab93841282c08ec5" candidate_uuid: "44cd59eb508e40f3af451ae1ef705aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027a66dda1f7407c9d4f64f01dbfd63b"
I20260926 05:50:40.538739 29398 raft_consensus.cc:3037] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:40.539505 29398 raft_consensus.cc:2454] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44cd59eb508e40f3af451ae1ef705aa4 in term 1.
I20260926 05:50:40.539708 29600 leader_election.cc:302] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027a66dda1f7407c9d4f64f01dbfd63b, 44cd59eb508e40f3af451ae1ef705aa4; no voters:
I20260926 05:50:40.539753 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f58147f7e04910ab93841282c08ec5" candidate_uuid: "44cd59eb508e40f3af451ae1ef705aa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e5dd7b39444bdeb6667d3309e0861d"
I20260926 05:50:40.539834 29736 raft_consensus.cc:2781] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:40.539844 29532 raft_consensus.cc:3037] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:40.540010 29736 raft_consensus.cc:686] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 44cd59eb508e40f3af451ae1ef705aa4, State: Running, Role: LEADER
I20260926 05:50:40.540140 29736 consensus_queue.cc:237] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } }
I20260926 05:50:40.540786 29532 raft_consensus.cc:2454] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44cd59eb508e40f3af451ae1ef705aa4 in term 1.
I20260926 05:50:40.541193 29253 catalog_manager.cc:5495] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44cd59eb508e40f3af451ae1ef705aa4 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "44cd59eb508e40f3af451ae1ef705aa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cd59eb508e40f3af451ae1ef705aa4" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 } health_report { overall_health: UNKNOWN } } }
W20260926 05:50:40.563974 29713 tablet.cc:2369] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:50:40.579954 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.577358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.11.247.3:46813)
0926 05:50:40.577431 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.577433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.577444 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:50:40.577517 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.577518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.577537 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.577614 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:40.577752 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.578258 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.579074 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.579310 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.579393 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.579573 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:40.579682 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:40.579780 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.579803 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.579825 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:50:40.581986 29532 raft_consensus.cc:1261] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 1 FOLLOWER]: Refusing update from remote peer 44cd59eb508e40f3af451ae1ef705aa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:40.582084 29398 raft_consensus.cc:1261] T 08f58147f7e04910ab93841282c08ec5 P 027a66dda1f7407c9d4f64f01dbfd63b [term 1 FOLLOWER]: Refusing update from remote peer 44cd59eb508e40f3af451ae1ef705aa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:40.582350 29736 consensus_queue.cc:1035] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e5dd7b39444bdeb6667d3309e0861d" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:40.582540 29739 consensus_queue.cc:1035] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "027a66dda1f7407c9d4f64f01dbfd63b" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:40.585223 29744 mvcc.cc:205] Tried to move back new op lower bound from 7333485939020767232 to 7333485938853388288. Current Snapshot: MvccSnapshot[applied={T|T < 7333485939020767232}]
I20260926 05:50:40.588094 29253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:40.588191 29253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:40.589021 29253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd5042cc5b734e85961bdbef56b9e202: Sending DeleteTablet for 3 replicas of tablet 08f58147f7e04910ab93841282c08ec5
I20260926 05:50:40.589337 29646 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f58147f7e04910ab93841282c08ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:40 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:39452
I20260926 05:50:40.589457 29512 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f58147f7e04910ab93841282c08ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:40 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:38616
I20260926 05:50:40.589533 29378 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f58147f7e04910ab93841282c08ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:40 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:39988
I20260926 05:50:40.589685 29752 tablet_replica.cc:330] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: stopping tablet replica
I20260926 05:50:40.589838 29752 raft_consensus.cc:2229] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:50:40.589985 29752 raft_consensus.cc:2258] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:40.590034 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29315
I20260926 05:50:40.593134 29752 ts_tablet_manager.cc:1905] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:40.594784 29753 tablet_replica.cc:330] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: stopping tablet replica
I20260926 05:50:40.595036 29753 raft_consensus.cc:2229] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:40.595232 29753 raft_consensus.cc:2258] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:40.595702 29752 ts_tablet_manager.cc:1918] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:40.595796 29752 log.cc:1196] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/08f58147f7e04910ab93841282c08ec5
I20260926 05:50:40.595925 29753 ts_tablet_manager.cc:1905] T 08f58147f7e04910ab93841282c08ec5 P 87e5dd7b39444bdeb6667d3309e0861d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:40.596092 29752 ts_tablet_manager.cc:1939] T 08f58147f7e04910ab93841282c08ec5 P 44cd59eb508e40f3af451ae1ef705aa4: Deleting consensus metadata
W20260926 05:50:40.596416 29238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:33755 (error 108)
I20260926 05:50:40.596518 29239 catalog_manager.cc:4841] TS 44cd59eb508e40f3af451ae1ef705aa4 (127.11.247.3:46813): tablet 08f58147f7e04910ab93841282c08ec5 (table test-table [id=bc35a3407ff8439f8d40c12c54813cc4]) successfully deleted
I20260926 05:50:40.597013 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29449
I20260926 05:50:40.599408 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.599225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:33755 (local address 127.0.0.1:40002)
0926 05:50:40.599301 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.599360 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:33755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 05:50:40.599609 29238 catalog_manager.cc:4584] TS 027a66dda1f7407c9d4f64f01dbfd63b (127.11.247.1:33755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08f58147f7e04910ab93841282c08ec5: Network error: Client connection negotiation failed: client connection to 127.11.247.1:33755: connect: Connection refused (error 111)
I20260926 05:50:40.602476 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29583
I20260926 05:50:40.602557 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.602307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35847 (local address 127.0.0.1:38628)
0926 05:50:40.602491 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.602519 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.2:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 05:50:40.607167 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29221
2026-09-26T05:50:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2144 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T05:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:40Z Disabled control of system clock
I20260926 05:50:40.636698 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33543
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:40023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:40.720034 29759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:40.720309 29759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:40.720345 29759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:40.721576 29759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:40.721655 29759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:40.721675 29759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:40.721696 29759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:40.721737 29759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:40.723256 29759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:40.723570 29759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:40.723893 29759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:40.729058 29765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.729027 29767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.729031 29764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:40.729449 29759 server_base.cc:1034] running on GCE node
I20260926 05:50:40.729738 29759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:40.730052 29759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:40.731228 29759 hybrid_clock.cc:648] HybridClock initialized: now 1790401840731218 us; error 38 us; skew 500 ppm
I20260926 05:50:40.732654 29759 webserver.cc:460] Webserver started at http://127.11.247.62:36825/ using document root <none> and password file <none>
I20260926 05:50:40.732939 29759 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:40.733016 29759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:40.733173 29759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:40.734313 29759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "8d6af45661ca4e649b1acf225ebd69c7"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "b5e0fe8cdd656d4dbfc1323721d25f48"
server_key_iv: "40cd1ee9be9aa91d49184f444d010ec4"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.734745 29759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "8d6af45661ca4e649b1acf225ebd69c7"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "b5e0fe8cdd656d4dbfc1323721d25f48"
server_key_iv: "40cd1ee9be9aa91d49184f444d010ec4"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.736096 29759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:40.737010 29773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.737207 29759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:40.737310 29759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "8d6af45661ca4e649b1acf225ebd69c7"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "b5e0fe8cdd656d4dbfc1323721d25f48"
server_key_iv: "40cd1ee9be9aa91d49184f444d010ec4"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.737397 29759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:40.755327 29759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:40.755695 29759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:40.755867 29759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:40.760569 29759 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33543
I20260926 05:50:40.760596 29825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33543 every 8 connection(s)
I20260926 05:50:40.760990 29759 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:40.761646 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29759
I20260926 05:50:40.761765 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:40.762006 12252 external_mini_cluster.cc:1428] Setting key 9fcad4a6f74f476795eb181d0bf87562
I20260926 05:50:40.762300 29826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:40.765318 29826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7: Bootstrap starting.
I20260926 05:50:40.765959 29826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:40.766263 29826 log.cc:824] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:40.767040 29826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7: No bootstrap required, opened a new log
I20260926 05:50:40.767800 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.762660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.11.247.62:33543)
0926 05:50:40.762916 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.762923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.762954 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:50:40.763500 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.763507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.763574 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.763732 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:40.763927 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.764866 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.765774 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.766270 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.766408 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.767163 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:40.767168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:40.767168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:40.767343 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:40.767375 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:40.767386 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:40.767465 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.767544 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.767629 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":136,"threads_started":1}
I20260926 05:50:40.769138 29826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6af45661ca4e649b1acf225ebd69c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33543 } }
I20260926 05:50:40.769309 29826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:40.769359 29826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6af45661ca4e649b1acf225ebd69c7, State: Initialized, Role: FOLLOWER
I20260926 05:50:40.769488 29826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6af45661ca4e649b1acf225ebd69c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33543 } }
I20260926 05:50:40.769572 29826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:40.769616 29826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:40.769667 29826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:40.770232 29826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6af45661ca4e649b1acf225ebd69c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33543 } }
I20260926 05:50:40.770359 29826 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6af45661ca4e649b1acf225ebd69c7; no voters:
I20260926 05:50:40.770553 29826 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:40.770650 29831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:40.770823 29831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6af45661ca4e649b1acf225ebd69c7, State: Running, Role: LEADER
I20260926 05:50:40.770877 29826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:40.770918 29831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6af45661ca4e649b1acf225ebd69c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33543 } }
I20260926 05:50:40.771287 29833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6af45661ca4e649b1acf225ebd69c7. Latest consensus state: current_term: 1 leader_uuid: "8d6af45661ca4e649b1acf225ebd69c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6af45661ca4e649b1acf225ebd69c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33543 } } }
I20260926 05:50:40.771296 29832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6af45661ca4e649b1acf225ebd69c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6af45661ca4e649b1acf225ebd69c7" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33543 } } }
I20260926 05:50:40.771402 29833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:40.771416 29832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:40.771610 29838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:40.772113 29838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:40.773609 29838 catalog_manager.cc:1347] Generated new cluster ID: 2cd20af4ff394349a4bb66dd731e6354
I20260926 05:50:40.773663 29838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:40.785753 29838 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:40.786299 29838 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:40.791956 29838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7: Generated new TSK 0
I20260926 05:50:40.792160 29838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:40.801828 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33543
--builtin_ntp_servers=127.11.247.20:40023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:40.889829 29850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:40.890059 29850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:40.890084 29850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:40.891417 29850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:40.891467 29850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:40.891496 29850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:40.892944 29850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:40.893232 29850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:40.893577 29850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:40.898767 29858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.898797 29855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:40.898767 29856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:40.899220 29850 server_base.cc:1034] running on GCE node
I20260926 05:50:40.899443 29850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:40.899698 29850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:40.900890 29850 hybrid_clock.cc:648] HybridClock initialized: now 1790401840900876 us; error 31 us; skew 500 ppm
I20260926 05:50:40.908113 29850 webserver.cc:460] Webserver started at http://127.11.247.1:46103/ using document root <none> and password file <none>
I20260926 05:50:40.908471 29850 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:40.908560 29850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:40.908723 29850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:40.909876 29850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "92070f0e7a254edc9fbd3cb74a5bf50a"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "6b063451588e1c92e3baef5078d99ba8"
server_key_iv: "85c4ecbbbd104cdaff7507dbef429e01"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.910298 29850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "92070f0e7a254edc9fbd3cb74a5bf50a"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "6b063451588e1c92e3baef5078d99ba8"
server_key_iv: "85c4ecbbbd104cdaff7507dbef429e01"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.911834 29850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:40.912731 29864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.912971 29850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:40.913100 29850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "92070f0e7a254edc9fbd3cb74a5bf50a"
format_stamp: "Formatted at 2026-09-26 05:50:40 on dist-test-slave-21p4"
server_key: "6b063451588e1c92e3baef5078d99ba8"
server_key_iv: "85c4ecbbbd104cdaff7507dbef429e01"
server_key_version: "encryptionkey@0"
I20260926 05:50:40.913184 29850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:40.923440 29850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:40.923776 29850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:40.923933 29850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:40.924183 29850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:40.924530 29850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:40.924585 29850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.924629 29850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:40.924654 29850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:40.930804 29850 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:36583
I20260926 05:50:40.930855 29977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:36583 every 8 connection(s)
I20260926 05:50:40.931200 29850 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:40.936856 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.932036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:55861 (local address 127.11.247.62:33543)
0926 05:50:40.932154 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:40.932183 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:40.932389 (+ 206us) server_negotiation.cc:407] Connection header received
0926 05:50:40.932950 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:40.932951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:40.932970 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:40.933036 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:40.933729 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.934282 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.935380 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.935606 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.935688 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.936531 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:40.936533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:40.936533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:40.936539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:40.936554 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:40.936559 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:40.936611 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:40.936740 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 05:50:40.936772 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 05:50:40.937114 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:40.931992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33543 (local address 127.11.247.1:55861)
0926 05:50:40.932262 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:40.932298 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:40.932832 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:40.933108 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:40.933112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:40.933120 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:40.933637 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.933640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.934387 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.934388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.935284 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:40.935286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:40.935697 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:40.935698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:40.935832 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:40.936375 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:40.936376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:40.936405 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:40.936429 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:40.936624 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:40.936629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:40.936632 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:40.936741 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:50:40.936850 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260926 05:50:40.937705 29978 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33543
I20260926 05:50:40.937788 29978 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:40.937996 29978 heartbeater.cc:511] Master 127.11.247.62:33543 requested a full tablet report, sending...
I20260926 05:50:40.938552 29790 ts_manager.cc:194] Registered new tserver with Master: 92070f0e7a254edc9fbd3cb74a5bf50a (127.11.247.1:36583)
I20260926 05:50:40.939254 29790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.1:55861
I20260926 05:50:40.940065 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29850
I20260926 05:50:40.940182 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:50:40.940385 12252 external_mini_cluster.cc:1428] Setting key 412c1e7b72a436b8c990c57a52f3b182
I20260926 05:50:40.941293 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33543
--builtin_ntp_servers=127.11.247.20:40023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:41.026180 29981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:41.026408 29981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:41.026433 29981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:41.027714 29981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:41.027765 29981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:41.027793 29981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:41.029297 29981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:41.029588 29981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:41.030015 29981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:41.035250 29987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:41.035254 29989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:41.035252 29986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:41.035861 29981 server_base.cc:1034] running on GCE node
I20260926 05:50:41.036119 29981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:41.036402 29981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:41.037569 29981 hybrid_clock.cc:648] HybridClock initialized: now 1790401841037563 us; error 40 us; skew 500 ppm
I20260926 05:50:41.039006 29981 webserver.cc:460] Webserver started at http://127.11.247.2:34721/ using document root <none> and password file <none>
I20260926 05:50:41.039294 29981 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:41.039367 29981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:41.039551 29981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:41.040762 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "1a208dd6f53e434885331be546e7e922"
format_stamp: "Formatted at 2026-09-26 05:50:41 on dist-test-slave-21p4"
server_key: "bbe6591b5f1fe9d956676e12e26b85a5"
server_key_iv: "b1d75a03ff7948364e9375da138ad61a"
server_key_version: "encryptionkey@0"
I20260926 05:50:41.041359 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "1a208dd6f53e434885331be546e7e922"
format_stamp: "Formatted at 2026-09-26 05:50:41 on dist-test-slave-21p4"
server_key: "bbe6591b5f1fe9d956676e12e26b85a5"
server_key_iv: "b1d75a03ff7948364e9375da138ad61a"
server_key_version: "encryptionkey@0"
I20260926 05:50:41.043002 29981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260926 05:50:41.043932 29995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:41.044185 29981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:41.044323 29981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "1a208dd6f53e434885331be546e7e922"
format_stamp: "Formatted at 2026-09-26 05:50:41 on dist-test-slave-21p4"
server_key: "bbe6591b5f1fe9d956676e12e26b85a5"
server_key_iv: "b1d75a03ff7948364e9375da138ad61a"
server_key_version: "encryptionkey@0"
I20260926 05:50:41.044420 29981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:41.057343 29981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:41.057723 29981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:41.057914 29981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:41.058185 29981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:41.058547 29981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:41.058631 29981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:41.058696 29981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:41.058722 29981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:41.064841 29981 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:36469
I20260926 05:50:41.064893 30108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:36469 every 8 connection(s)
I20260926 05:50:41.065245 29981 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:50:41.066265 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 29981
I20260926 05:50:41.066371 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:50:41.066636 12252 external_mini_cluster.cc:1428] Setting key 91cc73317535c3f37c4d4438c841af8f
I20260926 05:50:41.067746 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33543
--builtin_ntp_servers=127.11.247.20:40023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:50:41.071645 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.066046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:57445 (local address 127.11.247.62:33543)
0926 05:50:41.066139 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.066143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.066504 (+ 361us) server_negotiation.cc:407] Connection header received
0926 05:50:41.067192 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.067193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.067204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.067284 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:41.068233 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.068777 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.069962 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.070201 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.070262 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.071329 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:41.071331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:41.071331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:41.071337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:41.071353 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:41.071357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:41.071439 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.071553 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.071579 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 05:50:41.071992 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.066032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33543 (local address 127.11.247.2:57445)
0926 05:50:41.066302 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.066391 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.067080 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.067369 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.067372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.067381 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:41.068134 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.068142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.068889 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.068891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.069836 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.069839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.070302 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.070304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.070489 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.071159 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:41.071160 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:41.071203 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:41.071236 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:41.071455 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:41.071463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:41.071467 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.071557 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.071696 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260926 05:50:41.072579 30109 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33543
I20260926 05:50:41.072701 30109 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:41.072943 30109 heartbeater.cc:511] Master 127.11.247.62:33543 requested a full tablet report, sending...
I20260926 05:50:41.073411 29790 ts_manager.cc:194] Registered new tserver with Master: 1a208dd6f53e434885331be546e7e922 (127.11.247.2:36469)
I20260926 05:50:41.073892 29790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.2:57445
W20260926 05:50:41.155433 30112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:41.155696 30112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:41.155735 30112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:41.156900 30112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:41.156967 30112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:41.157012 30112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:41.158407 30112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:41.158701 30112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:41.159029 30112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:41.164227 30120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:41.164242 30117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:41.164247 30118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:41.164565 30112 server_base.cc:1034] running on GCE node
I20260926 05:50:41.164784 30112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:41.165035 30112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:41.166234 30112 hybrid_clock.cc:648] HybridClock initialized: now 1790401841166213 us; error 46 us; skew 500 ppm
I20260926 05:50:41.167868 30112 webserver.cc:460] Webserver started at http://127.11.247.3:39805/ using document root <none> and password file <none>
I20260926 05:50:41.168143 30112 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:41.168223 30112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:41.168392 30112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:41.169564 30112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "e88487beff4e4946ab46a3de4a60e2cb"
format_stamp: "Formatted at 2026-09-26 05:50:41 on dist-test-slave-21p4"
server_key: "c11b76de1424d6a7cf7f65d49cb5ee07"
server_key_iv: "99cf57497ee3cb3d1ac567b6a2c7e754"
server_key_version: "encryptionkey@0"
I20260926 05:50:41.169975 30112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "e88487beff4e4946ab46a3de4a60e2cb"
format_stamp: "Formatted at 2026-09-26 05:50:41 on dist-test-slave-21p4"
server_key: "c11b76de1424d6a7cf7f65d49cb5ee07"
server_key_iv: "99cf57497ee3cb3d1ac567b6a2c7e754"
server_key_version: "encryptionkey@0"
I20260926 05:50:41.171787 30112 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 05:50:41.172708 30126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:41.172904 30112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:41.173012 30112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "e88487beff4e4946ab46a3de4a60e2cb"
format_stamp: "Formatted at 2026-09-26 05:50:41 on dist-test-slave-21p4"
server_key: "c11b76de1424d6a7cf7f65d49cb5ee07"
server_key_iv: "99cf57497ee3cb3d1ac567b6a2c7e754"
server_key_version: "encryptionkey@0"
I20260926 05:50:41.173090 30112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:41.192948 30112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:41.193356 30112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:41.193521 30112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:41.193780 30112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:41.194154 30112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:41.194209 30112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:41.194247 30112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:41.194278 30112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:41.200567 30112 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:32787
I20260926 05:50:41.200613 30239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:32787 every 8 connection(s)
I20260926 05:50:41.200971 30112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:50:41.203568 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30112
I20260926 05:50:41.203689 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:50:41.203913 12252 external_mini_cluster.cc:1428] Setting key eb315cf43e0efc8de5554ffeb69fc42d
I20260926 05:50:41.208874 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.201825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:59391 (local address 127.11.247.62:33543)
0926 05:50:41.201947 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.201949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.202156 (+ 207us) server_negotiation.cc:407] Connection header received
0926 05:50:41.204935 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.204936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.204951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.205001 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:41.205766 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.206290 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.207364 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.207569 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.207651 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.208556 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:41.208557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:41.208558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:41.208562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:41.208581 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:41.208590 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:41.208650 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.208763 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.208793 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 05:50:41.209127 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.201730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33543 (local address 127.11.247.3:59391)
0926 05:50:41.202015 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.202067 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.204799 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.205089 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.205092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.205101 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:41.205628 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.205640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.206392 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.206393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.207269 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.207271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.207659 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.207660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.207802 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.208385 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:41.208386 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:41.208415 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:41.208439 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:41.208651 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:41.208658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:41.208661 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.208763 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.208869 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
I20260926 05:50:41.209638 30240 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33543
I20260926 05:50:41.209738 30240 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:41.209964 30240 heartbeater.cc:511] Master 127.11.247.62:33543 requested a full tablet report, sending...
I20260926 05:50:41.210354 29790 ts_manager.cc:194] Registered new tserver with Master: e88487beff4e4946ab46a3de4a60e2cb (127.11.247.3:32787)
I20260926 05:50:41.210827 29790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.3:59391
I20260926 05:50:41.215727 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:41.220506 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.217254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.11.247.62:33543)
0926 05:50:41.217315 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.217318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.217481 (+ 163us) server_negotiation.cc:407] Connection header received
0926 05:50:41.217581 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.217583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.217594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.217650 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:41.217944 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.218416 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.219299 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.219491 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.219557 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.220272 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:41.220274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:41.220274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:41.220278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:41.220293 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:41.220297 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:41.220351 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.220418 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.220442 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:50:41.222275 29790 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:41.223958 29790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:41.231212 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.226496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:32787 (local address 127.0.0.1:41576)
0926 05:50:41.226726 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.226768 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.226836 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.227488 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.227489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.227491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:41.227570 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.227572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.228550 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.228551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.229947 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.229950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.230848 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.230849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.230930 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.230944 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.231039 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.231099 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260926 05:50:41.231271 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.226650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.11.247.2:36469)
0926 05:50:41.227266 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.227271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.227281 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:50:41.227320 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.227322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.227337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.227628 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:41.227692 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.228806 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.230207 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.230955 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.231035 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.231120 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.231148 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.231188 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":149,"threads_started":1}
I20260926 05:50:41.231355 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.226750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.11.247.1:36583)
0926 05:50:41.227013 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.227017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.227188 (+ 171us) server_negotiation.cc:407] Connection header received
0926 05:50:41.227237 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.227239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.227256 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.227343 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:41.227640 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.228827 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.229980 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.230729 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.230802 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.230901 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.231213 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.231268 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":155,"threads_started":1}
I20260926 05:50:41.231590 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.227015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.11.247.3:32787)
0926 05:50:41.227306 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.227310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.227321 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:50:41.227361 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.227364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.227380 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.227487 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:41.227629 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.228410 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.230051 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.230704 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.230840 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.230935 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.231261 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.231513 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260926 05:50:41.231786 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.226540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36583 (local address 127.0.0.1:37750)
0926 05:50:41.227056 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.227065 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.227143 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.227432 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.227436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.227438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:41.227558 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.227559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.228927 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.228928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.229882 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.229885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.230939 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.230940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.231007 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.231013 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.231061 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.231108 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":46,"thread_start_us":457,"threads_started":1}
I20260926 05:50:41.231786 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.226496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:36469 (local address 127.0.0.1:44622)
0926 05:50:41.226906 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.226918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.226983 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.227432 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.227436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.227440 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:41.227548 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.227551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.228929 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.228930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.230122 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.230125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.231036 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.231037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.231090 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.231094 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.231135 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.231169 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":30,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260926 05:50:41.232321 30174 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2f3e5697cf413e96cb03c3ef2e852c (DEFAULT_TABLE table=test-table [id=3d8f44f4d48f4520bd4b6625a5c30aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:41.232422 29912 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2f3e5697cf413e96cb03c3ef2e852c (DEFAULT_TABLE table=test-table [id=3d8f44f4d48f4520bd4b6625a5c30aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:41.232625 30174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2f3e5697cf413e96cb03c3ef2e852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:41.232687 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2f3e5697cf413e96cb03c3ef2e852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:41.234813 30043 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2f3e5697cf413e96cb03c3ef2e852c (DEFAULT_TABLE table=test-table [id=3d8f44f4d48f4520bd4b6625a5c30aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:41.235165 30043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2f3e5697cf413e96cb03c3ef2e852c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:41.235795 30255 tablet_bootstrap.cc:492] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a: Bootstrap starting.
I20260926 05:50:41.235909 30254 tablet_bootstrap.cc:492] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Bootstrap starting.
I20260926 05:50:41.236492 30255 tablet_bootstrap.cc:654] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:41.236500 30254 tablet_bootstrap.cc:654] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:41.236730 30254 log.cc:824] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:41.236841 30255 log.cc:824] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:41.237653 30254 tablet_bootstrap.cc:492] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: No bootstrap required, opened a new log
I20260926 05:50:41.237740 30254 ts_tablet_manager.cc:1397] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:50:41.239356 30258 tablet_bootstrap.cc:492] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Bootstrap starting.
I20260926 05:50:41.239748 30255 tablet_bootstrap.cc:492] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a: No bootstrap required, opened a new log
I20260926 05:50:41.239832 30255 ts_tablet_manager.cc:1397] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 05:50:41.240219 30258 tablet_bootstrap.cc:654] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:41.240566 30258 log.cc:824] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:41.240710 30254 raft_consensus.cc:348] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.240908 30254 raft_consensus.cc:374] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:41.240950 30254 raft_consensus.cc:729] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88487beff4e4946ab46a3de4a60e2cb, State: Initialized, Role: FOLLOWER
I20260926 05:50:41.241096 30254 consensus_queue.cc:260] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.241375 30254 ts_tablet_manager.cc:1428] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:50:41.241422 30240 heartbeater.cc:503] Master 127.11.247.62:33543 was elected leader, sending a full tablet report...
I20260926 05:50:41.241465 30258 tablet_bootstrap.cc:492] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: No bootstrap required, opened a new log
I20260926 05:50:41.241524 30258 ts_tablet_manager.cc:1397] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:50:41.242646 30255 raft_consensus.cc:348] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.242836 30255 raft_consensus.cc:374] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:41.242877 30255 raft_consensus.cc:729] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92070f0e7a254edc9fbd3cb74a5bf50a, State: Initialized, Role: FOLLOWER
I20260926 05:50:41.242998 30255 consensus_queue.cc:260] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.243346 30255 ts_tablet_manager.cc:1428] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:50:41.243402 29978 heartbeater.cc:503] Master 127.11.247.62:33543 was elected leader, sending a full tablet report...
I20260926 05:50:41.244287 30258 raft_consensus.cc:348] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.244493 30258 raft_consensus.cc:374] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:41.244539 30258 raft_consensus.cc:729] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a208dd6f53e434885331be546e7e922, State: Initialized, Role: FOLLOWER
I20260926 05:50:41.244669 30258 consensus_queue.cc:260] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.244964 30258 ts_tablet_manager.cc:1428] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:50:41.245121 30109 heartbeater.cc:503] Master 127.11.247.62:33543 was elected leader, sending a full tablet report...
W20260926 05:50:41.315912 30110 tablet.cc:2369] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:50:41.318763 30260 raft_consensus.cc:482] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:41.318909 30260 raft_consensus.cc:504] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.319240 30260 leader_election.cc:288] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a208dd6f53e434885331be546e7e922 (127.11.247.2:36469), 92070f0e7a254edc9fbd3cb74a5bf50a (127.11.247.1:36583)
I20260926 05:50:41.323777 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.319503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:39405 (local address 127.11.247.1:36583)
0926 05:50:41.319652 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.319655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.319824 (+ 169us) server_negotiation.cc:407] Connection header received
0926 05:50:41.319902 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.319903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.319916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.320004 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:41.320152 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.320898 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.322772 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.323353 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.323424 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.323433 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.323657 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.323700 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260926 05:50:41.323777 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.319709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:36331 (local address 127.11.247.2:36469)
0926 05:50:41.319793 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.319795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.319804 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:41.319833 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.319834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.319843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.319890 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:41.320125 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.320926 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.322778 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.323341 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.323424 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.323433 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.323657 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.323700 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 05:50:41.324054 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.319468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36583 (local address 127.11.247.3:39405)
0926 05:50:41.319728 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.319739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.319830 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.320014 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.320015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.320016 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:41.320073 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.320074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.321056 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.321057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.322660 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.322665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.323457 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.323459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.323558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.323566 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.323637 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.323718 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":96,"thread_start_us":189,"threads_started":1}
I20260926 05:50:41.324318 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.319450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:36469 (local address 127.11.247.3:36331)
0926 05:50:41.319565 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.319579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:41.319630 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:41.319946 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:41.319947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:41.319949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:41.320022 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.320024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.321056 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.321057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.322660 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:41.322665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.323457 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.323459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:41.323558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.323566 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 05:50:41.323637 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:50:41.323693 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":75}
I20260926 05:50:41.324460 29932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2f3e5697cf413e96cb03c3ef2e852c" candidate_uuid: "e88487beff4e4946ab46a3de4a60e2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" is_pre_election: true
I20260926 05:50:41.324640 29932 raft_consensus.cc:2454] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e88487beff4e4946ab46a3de4a60e2cb in term 0.
I20260926 05:50:41.324717 30063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2f3e5697cf413e96cb03c3ef2e852c" candidate_uuid: "e88487beff4e4946ab46a3de4a60e2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a208dd6f53e434885331be546e7e922" is_pre_election: true
I20260926 05:50:41.324908 30063 raft_consensus.cc:2454] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e88487beff4e4946ab46a3de4a60e2cb in term 0.
I20260926 05:50:41.324896 30127 leader_election.cc:302] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92070f0e7a254edc9fbd3cb74a5bf50a, e88487beff4e4946ab46a3de4a60e2cb; no voters:
I20260926 05:50:41.325096 30260 raft_consensus.cc:2781] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:41.325176 30260 raft_consensus.cc:482] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:41.325212 30260 raft_consensus.cc:3037] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:41.326009 30260 raft_consensus.cc:504] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.326175 30260 leader_election.cc:288] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [CANDIDATE]: Term 1 election: Requested vote from peers 1a208dd6f53e434885331be546e7e922 (127.11.247.2:36469), 92070f0e7a254edc9fbd3cb74a5bf50a (127.11.247.1:36583)
I20260926 05:50:41.326359 30063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2f3e5697cf413e96cb03c3ef2e852c" candidate_uuid: "e88487beff4e4946ab46a3de4a60e2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a208dd6f53e434885331be546e7e922"
I20260926 05:50:41.326372 29932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2f3e5697cf413e96cb03c3ef2e852c" candidate_uuid: "e88487beff4e4946ab46a3de4a60e2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a"
I20260926 05:50:41.326454 29932 raft_consensus.cc:3037] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:41.326454 30063 raft_consensus.cc:3037] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:41.327458 29932 raft_consensus.cc:2454] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e88487beff4e4946ab46a3de4a60e2cb in term 1.
I20260926 05:50:41.327458 30063 raft_consensus.cc:2454] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e88487beff4e4946ab46a3de4a60e2cb in term 1.
I20260926 05:50:41.327654 30130 leader_election.cc:302] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a208dd6f53e434885331be546e7e922, e88487beff4e4946ab46a3de4a60e2cb; no voters:
I20260926 05:50:41.327757 30260 raft_consensus.cc:2781] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:41.327924 30260 raft_consensus.cc:686] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 1 LEADER]: Becoming Leader. State: Replica: e88487beff4e4946ab46a3de4a60e2cb, State: Running, Role: LEADER
I20260926 05:50:41.328053 30260 consensus_queue.cc:237] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } }
I20260926 05:50:41.328994 29790 catalog_manager.cc:5495] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb reported cstate change: term changed from 0 to 1, leader changed from <none> to e88487beff4e4946ab46a3de4a60e2cb (127.11.247.3). New cstate: current_term: 1 leader_uuid: "e88487beff4e4946ab46a3de4a60e2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e88487beff4e4946ab46a3de4a60e2cb" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32787 } health_report { overall_health: HEALTHY } } }
I20260926 05:50:41.347347 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.344881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.11.247.3:32787)
0926 05:50:41.344942 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:41.344945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:41.344994 (+ 49us) server_negotiation.cc:407] Connection header received
0926 05:50:41.345058 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:41.345059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:41.345077 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:41.345141 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:41.345255 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.345741 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.346495 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:41.346761 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:41.346831 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:41.347024 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:41.347133 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:41.347211 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:41.347239 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:50:41.347257 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:50:41.349390 30063 raft_consensus.cc:1261] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 1 FOLLOWER]: Refusing update from remote peer e88487beff4e4946ab46a3de4a60e2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:41.349404 29932 raft_consensus.cc:1261] T 3c2f3e5697cf413e96cb03c3ef2e852c P 92070f0e7a254edc9fbd3cb74a5bf50a [term 1 FOLLOWER]: Refusing update from remote peer e88487beff4e4946ab46a3de4a60e2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:41.349766 30260 consensus_queue.cc:1035] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a208dd6f53e434885331be546e7e922" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:41.349929 30264 consensus_queue.cc:1035] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92070f0e7a254edc9fbd3cb74a5bf50a" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:41.352521 30267 mvcc.cc:205] Tried to move back new op lower bound from 7333485942164336640 to 7333485942080483328. Current Snapshot: MvccSnapshot[applied={T|T < 7333485942164336640}]
I20260926 05:50:41.353030 30269 mvcc.cc:205] Tried to move back new op lower bound from 7333485942164336640 to 7333485942080483328. Current Snapshot: MvccSnapshot[applied={T|T < 7333485942164336640}]
I20260926 05:50:41.355027 29790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:41.355127 29790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:41.356077 29790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d6af45661ca4e649b1acf225ebd69c7: Sending DeleteTablet for 3 replicas of tablet 3c2f3e5697cf413e96cb03c3ef2e852c
I20260926 05:50:41.356521 30043 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2f3e5697cf413e96cb03c3ef2e852c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:41 UTC) from {username='slave'} at 127.0.0.1:44622
I20260926 05:50:41.356776 29912 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2f3e5697cf413e96cb03c3ef2e852c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:41 UTC) from {username='slave'} at 127.0.0.1:37750
I20260926 05:50:41.356772 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29850
I20260926 05:50:41.357494 30277 tablet_replica.cc:330] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: stopping tablet replica
I20260926 05:50:41.357510 30174 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2f3e5697cf413e96cb03c3ef2e852c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:41 UTC) from {username='slave'} at 127.0.0.1:41576
I20260926 05:50:41.357707 30277 raft_consensus.cc:2229] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:41.357873 30277 raft_consensus.cc:2258] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:41.358325 30278 tablet_replica.cc:330] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: stopping tablet replica
I20260926 05:50:41.358423 30277 ts_tablet_manager.cc:1905] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:41.358520 30278 raft_consensus.cc:2229] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:50:41.358739 30278 raft_consensus.cc:2258] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:41.359354 30278 ts_tablet_manager.cc:1905] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:41.360721 30277 ts_tablet_manager.cc:1918] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:41.360816 30277 log.cc:1196] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/3c2f3e5697cf413e96cb03c3ef2e852c
I20260926 05:50:41.361117 30277 ts_tablet_manager.cc:1939] T 3c2f3e5697cf413e96cb03c3ef2e852c P 1a208dd6f53e434885331be546e7e922: Deleting consensus metadata
I20260926 05:50:41.361521 30278 ts_tablet_manager.cc:1918] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:41.361650 30278 log.cc:1196] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/3c2f3e5697cf413e96cb03c3ef2e852c
I20260926 05:50:41.362005 30278 ts_tablet_manager.cc:1939] T 3c2f3e5697cf413e96cb03c3ef2e852c P e88487beff4e4946ab46a3de4a60e2cb: Deleting consensus metadata
I20260926 05:50:41.362535 29776 catalog_manager.cc:4841] TS e88487beff4e4946ab46a3de4a60e2cb (127.11.247.3:32787): tablet 3c2f3e5697cf413e96cb03c3ef2e852c (table test-table [id=3d8f44f4d48f4520bd4b6625a5c30aa8]) successfully deleted
I20260926 05:50:41.362831 29777 catalog_manager.cc:4841] TS 1a208dd6f53e434885331be546e7e922 (127.11.247.2:36469): tablet 3c2f3e5697cf413e96cb03c3ef2e852c (table test-table [id=3d8f44f4d48f4520bd4b6625a5c30aa8]) successfully deleted
W20260926 05:50:41.364583 29774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:36583 (error 108)
I20260926 05:50:41.364774 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29981
I20260926 05:50:41.366331 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:41.365447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36583 (local address 127.0.0.1:37764)
0926 05:50:41.366254 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:41.366300 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782}
W20260926 05:50:41.366475 29774 catalog_manager.cc:4584] TS 92070f0e7a254edc9fbd3cb74a5bf50a (127.11.247.1:36583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c2f3e5697cf413e96cb03c3ef2e852c: Network error: Client connection negotiation failed: client connection to 127.11.247.1:36583: connect: Connection refused (error 111)
I20260926 05:50:41.371089 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30112
I20260926 05:50:41.375517 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 29759
2026-09-26T05:50:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (766 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:41 dist-test-slave-21p4 krb5kdc[30282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:41 dist-test-slave-21p4 krb5kdc[30282](info): set up 2 sockets
Sep 26 05:50:41 dist-test-slave-21p4 krb5kdc[30282](info): commencing operation
krb5kdc: starting...
W20260926 05:50:42.422736 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:42 dist-test-slave-21p4 krb5kdc[30282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:50:42.549171 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34035
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:42.631894 30295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:42.632107 30295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:42.632129 30295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:42.633309 30295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:42.633354 30295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:42.633369 30295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:42.633381 30295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:42.633394 30295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:42.634929 30295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:42.635187 30295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:42.635432 30295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:42.640326 30303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:42.640379 30300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:42.640525 30295 server_base.cc:1034] running on GCE node
W20260926 05:50:42.640349 30301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:42.641016 30295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:42.641352 30295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:42.642508 30295 hybrid_clock.cc:648] HybridClock initialized: now 1790401842642500 us; error 52 us; skew 500 ppm
Sep 26 05:50:42 dist-test-slave-21p4 krb5kdc[30282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:42.645040 30295 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:50:42.645435 30295 webserver.cc:460] Webserver started at http://127.11.247.62:35903/ using document root <none> and password file <none>
I20260926 05:50:42.645644 30295 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:42.645684 30295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:42.645776 30295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:42.646878 30295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "56678ec2db0d48d59e19c1b2c662c0d0"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "7a92af85c0f6868f8f185144fafd3022"
server_key_iv: "360a4c9e94375755ff107b06c3e918bc"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.647190 30295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "56678ec2db0d48d59e19c1b2c662c0d0"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "7a92af85c0f6868f8f185144fafd3022"
server_key_iv: "360a4c9e94375755ff107b06c3e918bc"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.648540 30295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:42.649281 30310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:42.649510 30295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:42.649600 30295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "56678ec2db0d48d59e19c1b2c662c0d0"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "7a92af85c0f6868f8f185144fafd3022"
server_key_iv: "360a4c9e94375755ff107b06c3e918bc"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.649662 30295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:42.669394 30295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:42.671473 30295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:42.671587 30295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:42.675654 30295 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:34035
I20260926 05:50:42.675740 30362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:34035 every 8 connection(s)
I20260926 05:50:42.676061 30295 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:42.677040 30363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:42.679348 30363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0: Bootstrap starting.
I20260926 05:50:42.679966 30363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:42.680266 30363 log.cc:824] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:42.681023 30363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0: No bootstrap required, opened a new log
I20260926 05:50:42.682753 30363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34035 } }
I20260926 05:50:42.682902 30363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:42.682925 30363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56678ec2db0d48d59e19c1b2c662c0d0, State: Initialized, Role: FOLLOWER
I20260926 05:50:42.683100 30363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34035 } }
I20260926 05:50:42.683166 30363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:42.683187 30363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:42.683238 30363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:42.683799 30363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34035 } }
I20260926 05:50:42.683885 30363 leader_election.cc:302] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56678ec2db0d48d59e19c1b2c662c0d0; no voters:
I20260926 05:50:42.684115 30363 leader_election.cc:288] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:42.684209 30366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:42.684381 30366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [term 1 LEADER]: Becoming Leader. State: Replica: 56678ec2db0d48d59e19c1b2c662c0d0, State: Running, Role: LEADER
I20260926 05:50:42.684413 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30295
I20260926 05:50:42.684442 30363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:42.684537 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:42.684507 30366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34035 } }
I20260926 05:50:42.684788 12252 external_mini_cluster.cc:1428] Setting key 50b885afeadcaca5a5327b6ed0d71a08
I20260926 05:50:42.684903 30367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34035 } } }
I20260926 05:50:42.684989 30367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:42.685140 30368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56678ec2db0d48d59e19c1b2c662c0d0. Latest consensus state: current_term: 1 leader_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56678ec2db0d48d59e19c1b2c662c0d0" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 34035 } } }
I20260926 05:50:42.685220 30374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:42.685251 30368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:42.685788 30374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:42.687358 30374 catalog_manager.cc:1347] Generated new cluster ID: 8da71d9d301b40d6ab7ba71cc37d1a4b
I20260926 05:50:42.687414 30374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:50:42 dist-test-slave-21p4 krb5kdc[30282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:42.699802 30374 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:42.700549 30374 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:42.700793 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:42.685494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.11.247.62:34035)
0926 05:50:42.685676 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:42.685687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:42.685728 (+ 41us) server_negotiation.cc:407] Connection header received
0926 05:50:42.685860 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:42.685867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:42.685934 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:42.686865 (+ 931us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:42.686992 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:42.691913 (+ 4921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:42.693068 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:42.693695 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:42.693797 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:42.696768 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:42.696773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:42.696774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:42.696788 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:42.698915 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:42.699347 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:42.699355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:42.699356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:42.699483 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:42.699700 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:42.699701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:42.699701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:42.700050 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:42.700209 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:42.700329 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 05:50:42.700545 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260926 05:50:42.704931 30374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0: Generated new TSK 0
I20260926 05:50:42.705127 30374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:50:42.738538 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34035
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:42.823926 30389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:42.824179 30389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:42.824221 30389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:42.825403 30389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:42.825474 30389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:42.825519 30389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:42.827066 30389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:42.827366 30389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:42.827606 30389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:42.832865 30397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:42.832818 30394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:42.832818 30395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:42.833096 30389 server_base.cc:1034] running on GCE node
I20260926 05:50:42.833317 30389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:42.833588 30389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:42.834764 30389 hybrid_clock.cc:648] HybridClock initialized: now 1790401842834741 us; error 39 us; skew 500 ppm
Sep 26 05:50:42 dist-test-slave-21p4 krb5kdc[30282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:42.837499 30389 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:50:42.837983 30389 webserver.cc:460] Webserver started at http://127.11.247.1:33311/ using document root <none> and password file <none>
I20260926 05:50:42.838186 30389 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:42.838233 30389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:42.838330 30389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:42.839432 30389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "13640ac89a134ab6a7bcf832c19d75a9"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "2831dec3dbf72aed5b9822b656a9dd24"
server_key_iv: "327f11bbd1badafd311ddf02442a533b"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.839753 30389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "13640ac89a134ab6a7bcf832c19d75a9"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "2831dec3dbf72aed5b9822b656a9dd24"
server_key_iv: "327f11bbd1badafd311ddf02442a533b"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.841048 30389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:50:42.841809 30404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:42.842032 30389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:42.842120 30389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "13640ac89a134ab6a7bcf832c19d75a9"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "2831dec3dbf72aed5b9822b656a9dd24"
server_key_iv: "327f11bbd1badafd311ddf02442a533b"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.842180 30389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:42.850574 30389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:42.852427 30389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:42.852609 30389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:42.852804 30389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:42.853068 30389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:42.853096 30389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:42.853119 30389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:42.853132 30389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:42.858757 30389 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:39847
I20260926 05:50:42.858834 30517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:39847 every 8 connection(s)
I20260926 05:50:42.859129 30389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:42.864075 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30389
I20260926 05:50:42.864205 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:50:42.864464 12252 external_mini_cluster.cc:1428] Setting key 021bf4e9f1dd00c771b2089c7c83f70e
Sep 26 05:50:42 dist-test-slave-21p4 krb5kdc[30282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:42.869339 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:42.859889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:35305 (local address 127.11.247.62:34035)
0926 05:50:42.859939 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:42.859942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:42.860140 (+ 198us) server_negotiation.cc:407] Connection header received
0926 05:50:42.860634 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:42.860635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:42.860645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:42.860705 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:42.861728 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:42.862278 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:42.863378 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:42.863592 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:42.863663 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:42.866076 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:42.866077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:42.866078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:42.866086 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:42.867648 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:42.868179 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:42.868180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:42.868180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:42.868257 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:42.868603 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:42.868604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:42.868604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:42.868722 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:42.868782 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:42.869179 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 05:50:42.869270 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:50:42.869594 30520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:42.859823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34035 (local address 127.11.247.1:35305)
0926 05:50:42.860001 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:42.860045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:42.860540 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:42.860775 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:42.860778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:42.861426 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:42.861624 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:42.861629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:42.862382 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:42.862383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:42.863269 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:42.863272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:42.863684 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:42.863685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:42.863810 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:42.864488 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:42.864497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:42.865977 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:42.867735 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:42.867744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:42.867746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:42.868077 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:42.868355 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:42.868356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:42.868356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:42.868520 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:42.868798 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:42.868806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:42.868998 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 05:50:42.869133 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 05:50:42.869337 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260926 05:50:42.870144 30518 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34035
I20260926 05:50:42.870311 30518 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:42.870539 30518 heartbeater.cc:511] Master 127.11.247.62:34035 requested a full tablet report, sending...
I20260926 05:50:42.871301 30327 ts_manager.cc:194] Registered new tserver with Master: 13640ac89a134ab6a7bcf832c19d75a9 (127.11.247.1:39847)
I20260926 05:50:42.871994 30327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:35305
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:50:42.893059 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34035
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:42.976926 30523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:42.977159 30523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:42.977185 30523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:42.978282 30523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:42.978320 30523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:42.978348 30523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:42.979972 30523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:42.980176 30523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:42.980371 30523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:42.985656 30529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:42.985621 30531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:42.985641 30528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:42.985970 30523 server_base.cc:1034] running on GCE node
I20260926 05:50:42.986182 30523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:42.986443 30523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:42.987634 30523 hybrid_clock.cc:648] HybridClock initialized: now 1790401842987619 us; error 25 us; skew 500 ppm
Sep 26 05:50:42 dist-test-slave-21p4 krb5kdc[30282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:42.990245 30523 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:50:42.990844 30523 webserver.cc:460] Webserver started at http://127.11.247.2:40379/ using document root <none> and password file <none>
I20260926 05:50:42.991077 30523 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:42.991147 30523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:42.991312 30523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:42.992400 30523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "abe0d24fd75342e091717bca24c03b60"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "fdfd631a99f4d5f8fcda1a9f6629f993"
server_key_iv: "0cccfefc78ef1e7423b4b8ce9749ed78"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.992770 30523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "abe0d24fd75342e091717bca24c03b60"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "fdfd631a99f4d5f8fcda1a9f6629f993"
server_key_iv: "0cccfefc78ef1e7423b4b8ce9749ed78"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.994177 30523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:42.995036 30538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:42.995224 30523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:50:42.995342 30523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "abe0d24fd75342e091717bca24c03b60"
format_stamp: "Formatted at 2026-09-26 05:50:42 on dist-test-slave-21p4"
server_key: "fdfd631a99f4d5f8fcda1a9f6629f993"
server_key_iv: "0cccfefc78ef1e7423b4b8ce9749ed78"
server_key_version: "encryptionkey@0"
I20260926 05:50:42.995426 30523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:43.015753 30523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:43.017891 30523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:43.018074 30523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:43.018352 30523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:43.018735 30523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:43.018792 30523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.018829 30523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:43.018875 30523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.025017 30523 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:37255
I20260926 05:50:43.025059 30651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:37255 every 8 connection(s)
I20260926 05:50:43.025413 30523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:50:43.028506 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30523
I20260926 05:50:43.028620 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:50:43.028855 12252 external_mini_cluster.cc:1428] Setting key d7d74930b3deffd2d6f030b54c03d3b9
Sep 26 05:50:43 dist-test-slave-21p4 krb5kdc[30282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:43.036386 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.026243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:53303 (local address 127.11.247.62:34035)
0926 05:50:43.026309 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.026313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.026472 (+ 159us) server_negotiation.cc:407] Connection header received
0926 05:50:43.026983 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.026984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.026997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.027059 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:43.027986 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.028690 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.030038 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.030365 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.030440 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.033316 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:43.033318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:43.033318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:43.033333 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:43.034921 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:43.035367 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:43.035369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:43.035369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:43.035430 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:43.035691 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:43.035692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:43.035692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:43.035830 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:43.035906 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.036197 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.036285 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:50:43.036595 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.026191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34035 (local address 127.11.247.2:53303)
0926 05:50:43.026349 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.026393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.026891 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.027107 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.027110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.027672 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:43.027872 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.027876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.028842 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.028843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.029945 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.029949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.030475 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.030483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.030665 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.031268 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:43.031280 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:43.033187 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:43.035024 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:43.035032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:43.035035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:43.035273 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:43.035501 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:43.035502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:43.035502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:43.035619 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:43.035908 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:43.035916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:43.036059 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.036175 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.036349 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260926 05:50:43.037135 30652 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34035
I20260926 05:50:43.037245 30652 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:43.037434 30652 heartbeater.cc:511] Master 127.11.247.62:34035 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:50:43.037843 30327 ts_manager.cc:194] Registered new tserver with Master: abe0d24fd75342e091717bca24c03b60 (127.11.247.2:37255)
I20260926 05:50:43.038187 30327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:53303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:50:43.056221 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34035
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:40031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:43.139125 30657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:43.139344 30657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:43.139369 30657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:43.140542 30657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:43.140586 30657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:43.140614 30657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:43.142143 30657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:40031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:43.142354 30657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:43.142657 30657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:43.147933 30665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:43.147954 30663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:43.148118 30657 server_base.cc:1034] running on GCE node
W20260926 05:50:43.148015 30662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:43.148425 30657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:43.148698 30657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:43.149895 30657 hybrid_clock.cc:648] HybridClock initialized: now 1790401843149882 us; error 48 us; skew 500 ppm
Sep 26 05:50:43 dist-test-slave-21p4 krb5kdc[30282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401843, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:43.152616 30657 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:50:43.153146 30657 webserver.cc:460] Webserver started at http://127.11.247.3:34319/ using document root <none> and password file <none>
I20260926 05:50:43.153354 30657 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:43.153398 30657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:43.153486 30657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:43.154487 30657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "9be9f573e8694c37abfe05204bb7dee5"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "ddc8e4f2f6ffeaab3c2684d83e4d87cd"
server_key_iv: "e4a8e246438d4b2504ff67650efac05c"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.154865 30657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "9be9f573e8694c37abfe05204bb7dee5"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "ddc8e4f2f6ffeaab3c2684d83e4d87cd"
server_key_iv: "e4a8e246438d4b2504ff67650efac05c"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.156185 30657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:43.156924 30672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.157109 30657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:43.157207 30657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "9be9f573e8694c37abfe05204bb7dee5"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "ddc8e4f2f6ffeaab3c2684d83e4d87cd"
server_key_iv: "e4a8e246438d4b2504ff67650efac05c"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.157274 30657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:43.174126 30657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:43.176589 30657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:43.176708 30657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:43.176923 30657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:43.177356 30657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:43.177395 30657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.177420 30657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:43.177433 30657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.183295 30657 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:37025
I20260926 05:50:43.183372 30785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:37025 every 8 connection(s)
I20260926 05:50:43.183704 30657 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:50:43 dist-test-slave-21p4 krb5kdc[30282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401843, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:43.191679 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30657
I20260926 05:50:43.191812 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:50:43.192025 12252 external_mini_cluster.cc:1428] Setting key f7e2ced8dcd5c081160caef21467ade7
I20260926 05:50:43.194278 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.184516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:34845 (local address 127.11.247.62:34035)
0926 05:50:43.184622 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.184634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.184829 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:50:43.185530 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.185531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.185542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.185586 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:43.186736 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.187253 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.188283 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.188494 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.188550 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.191006 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:43.191007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:43.191007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:43.191015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:43.192872 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:43.193369 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:43.193370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:43.193370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:43.193424 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:43.193685 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:43.193685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:43.193686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:43.193832 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:43.193891 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.194148 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.194213 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 05:50:43.194566 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.184476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:34035 (local address 127.11.247.3:34845)
0926 05:50:43.184658 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.184731 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.185425 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.185643 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.185646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.186359 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:43.186574 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.186635 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.187351 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.187352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.188201 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.188203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.188572 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.188573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.188684 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.189174 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:43.189183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:43.190888 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:43.192966 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:43.192981 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:43.192983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:43.193298 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:43.193506 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:43.193507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:43.193508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:43.193625 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:43.193897 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:43.193901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:43.194033 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.194120 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.194309 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260926 05:50:43.195065 30786 heartbeater.cc:347] Connected to a master server at 127.11.247.62:34035
I20260926 05:50:43.195185 30786 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:43.195379 30786 heartbeater.cc:511] Master 127.11.247.62:34035 requested a full tablet report, sending...
I20260926 05:50:43.195799 30327 ts_manager.cc:194] Registered new tserver with Master: 9be9f573e8694c37abfe05204bb7dee5 (127.11.247.3:37025)
I20260926 05:50:43.196206 30327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:34845
I20260926 05:50:43.204072 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:43.212381 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.205594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.11.247.62:34035)
0926 05:50:43.205670 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.205673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.205761 (+ 88us) server_negotiation.cc:407] Connection header received
0926 05:50:43.205846 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.205846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.205856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.205930 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:43.206549 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.207098 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.207846 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.208057 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.208137 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.209575 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:43.209576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:43.209577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:43.209585 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:43.211069 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:43.211503 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:43.211504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:43.211505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:43.211567 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:43.211793 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:43.211794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:43.211795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:43.211958 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:43.212077 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.212200 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.212264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:50:43.214082 30327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:43.215821 30327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:43.222477 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.218132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:39847 (local address 127.0.0.1:35644)
0926 05:50:43.218670 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.218678 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.218715 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.219055 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.219056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.219058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:43.219124 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.219125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.220090 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.220091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.221316 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.221318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.222148 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.222149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.222228 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.222243 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.222324 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.222388 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":20,"thread_start_us":112,"threads_started":1}
I20260926 05:50:43.222479 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.218569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.11.247.1:39847)
0926 05:50:43.218780 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.218784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.218797 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:50:43.218842 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.218844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.218863 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.218980 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:43.219194 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.219974 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.221411 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.222064 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.222122 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.222164 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.222343 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.222375 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":134,"threads_started":1}
I20260926 05:50:43.222852 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.218131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:37255 (local address 127.0.0.1:33712)
0926 05:50:43.218432 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.218465 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.218551 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.219025 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.219028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.219030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:43.219123 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.219125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.220509 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.220510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.221654 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.221657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.222427 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.222428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.222501 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.222505 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.222766 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.222801 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260926 05:50:43.223201 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.218251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.11.247.2:37255)
0926 05:50:43.218836 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.218840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.218849 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:43.218887 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.218890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.218904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.219205 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:43.219283 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.220398 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.221737 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.222347 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.222410 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.222464 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.223051 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.223096 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":77,"threads_started":1}
I20260926 05:50:43.223476 30452 tablet_service.cc:1467] Processing CreateTablet for tablet 61f79633561b493c86bf28ebeef23bf7 (DEFAULT_TABLE table=test-table [id=e35b51d3a90b4fa69d87d4558871809e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:43.223763 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.218134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:37025 (local address 127.0.0.1:49570)
0926 05:50:43.218759 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.218765 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.218819 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.219153 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.219154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.219156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:43.219218 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.219220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.220540 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.220541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.221642 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.221645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.223513 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.223514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.223584 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.223588 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.223634 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.223692 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260926 05:50:43.223793 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.218400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.11.247.3:37025)
0926 05:50:43.218575 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.218613 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.218821 (+ 208us) server_negotiation.cc:407] Connection header received
0926 05:50:43.218976 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.218979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.218997 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.219068 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:43.219305 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.220388 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.222503 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.223411 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.223512 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.223573 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.223671 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.223716 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260926 05:50:43.223873 30452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f79633561b493c86bf28ebeef23bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:43.226420 30803 tablet_bootstrap.cc:492] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: Bootstrap starting.
I20260926 05:50:43.226902 30586 tablet_service.cc:1467] Processing CreateTablet for tablet 61f79633561b493c86bf28ebeef23bf7 (DEFAULT_TABLE table=test-table [id=e35b51d3a90b4fa69d87d4558871809e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:43.227178 30803 tablet_bootstrap.cc:654] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:43.227145 30720 tablet_service.cc:1467] Processing CreateTablet for tablet 61f79633561b493c86bf28ebeef23bf7 (DEFAULT_TABLE table=test-table [id=e35b51d3a90b4fa69d87d4558871809e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:43.227262 30586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f79633561b493c86bf28ebeef23bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:43.227444 30720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f79633561b493c86bf28ebeef23bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:43.227752 30803 log.cc:824] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:43.228516 30803 tablet_bootstrap.cc:492] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: No bootstrap required, opened a new log
I20260926 05:50:43.228601 30803 ts_tablet_manager.cc:1397] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:50:43.230319 30803 raft_consensus.cc:348] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.230502 30803 raft_consensus.cc:374] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:43.230549 30803 raft_consensus.cc:729] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13640ac89a134ab6a7bcf832c19d75a9, State: Initialized, Role: FOLLOWER
I20260926 05:50:43.230716 30803 consensus_queue.cc:260] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.230772 30806 tablet_bootstrap.cc:492] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Bootstrap starting.
I20260926 05:50:43.230887 30805 tablet_bootstrap.cc:492] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Bootstrap starting.
I20260926 05:50:43.231012 30803 ts_tablet_manager.cc:1428] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:50:43.231138 30518 heartbeater.cc:503] Master 127.11.247.62:34035 was elected leader, sending a full tablet report...
I20260926 05:50:43.231583 30805 tablet_bootstrap.cc:654] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:43.231604 30806 tablet_bootstrap.cc:654] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:43.232158 30806 log.cc:824] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:43.232282 30805 log.cc:824] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:43.232971 30806 tablet_bootstrap.cc:492] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: No bootstrap required, opened a new log
I20260926 05:50:43.233027 30806 ts_tablet_manager.cc:1397] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:50:43.233180 30805 tablet_bootstrap.cc:492] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: No bootstrap required, opened a new log
I20260926 05:50:43.233296 30805 ts_tablet_manager.cc:1397] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:50:43.234869 30806 raft_consensus.cc:348] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.235024 30806 raft_consensus.cc:374] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:43.235046 30806 raft_consensus.cc:729] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe0d24fd75342e091717bca24c03b60, State: Initialized, Role: FOLLOWER
I20260926 05:50:43.235181 30806 consensus_queue.cc:260] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.235177 30805 raft_consensus.cc:348] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.235373 30805 raft_consensus.cc:374] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:43.235419 30805 raft_consensus.cc:729] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be9f573e8694c37abfe05204bb7dee5, State: Initialized, Role: FOLLOWER
I20260926 05:50:43.235488 30652 heartbeater.cc:503] Master 127.11.247.62:34035 was elected leader, sending a full tablet report...
I20260926 05:50:43.235496 30806 ts_tablet_manager.cc:1428] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:50:43.235543 30805 consensus_queue.cc:260] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.235841 30805 ts_tablet_manager.cc:1428] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:50:43.235852 30786 heartbeater.cc:503] Master 127.11.247.62:34035 was elected leader, sending a full tablet report...
W20260926 05:50:43.276087 30653 tablet.cc:2369] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:50:43.360085 30519 tablet.cc:2369] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:50:43.434420 30787 tablet.cc:2369] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:50:43.539662 30807 raft_consensus.cc:482] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:43.539803 30807 raft_consensus.cc:504] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.540285 30807 leader_election.cc:288] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe0d24fd75342e091717bca24c03b60 (127.11.247.2:37255), 9be9f573e8694c37abfe05204bb7dee5 (127.11.247.3:37025)
I20260926 05:50:43.543460 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.540606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:55083 (local address 127.11.247.3:37025)
0926 05:50:43.540660 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.540663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.540758 (+ 95us) server_negotiation.cc:407] Connection header received
0926 05:50:43.540836 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.540837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.540848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.540902 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:43.541086 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.541599 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.542684 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.543069 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.543156 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.543166 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.543338 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.543385 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:50:43.543479 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.540591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:42989 (local address 127.11.247.2:37255)
0926 05:50:43.540660 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.540663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.540753 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:50:43.540901 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.540902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.540910 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.540997 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:43.541174 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.541656 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.542683 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.543078 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.543165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.543173 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.543362 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.543413 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:50:43.543493 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.540518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:37025 (local address 127.11.247.1:55083)
0926 05:50:43.540678 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.540689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.540774 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.540925 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.540926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.540928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:43.541011 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.541013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.541696 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.541697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.542539 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.542542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.543166 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.543167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.543246 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.543251 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.543318 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.543366 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":20,"thread_start_us":80,"threads_started":1}
I20260926 05:50:43.543494 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.540512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:37255 (local address 127.11.247.1:42989)
0926 05:50:43.540658 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.540671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.540771 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.541005 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.541006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.541008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:43.541099 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.541101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.541746 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.541747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.542541 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.542544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.543163 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.543164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.543244 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.543250 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.543305 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.543363 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260926 05:50:43.543969 30740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f79633561b493c86bf28ebeef23bf7" candidate_uuid: "13640ac89a134ab6a7bcf832c19d75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be9f573e8694c37abfe05204bb7dee5" is_pre_election: true
I20260926 05:50:43.544140 30740 raft_consensus.cc:2454] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13640ac89a134ab6a7bcf832c19d75a9 in term 0.
I20260926 05:50:43.544100 30606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f79633561b493c86bf28ebeef23bf7" candidate_uuid: "13640ac89a134ab6a7bcf832c19d75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe0d24fd75342e091717bca24c03b60" is_pre_election: true
I20260926 05:50:43.544266 30606 raft_consensus.cc:2454] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13640ac89a134ab6a7bcf832c19d75a9 in term 0.
I20260926 05:50:43.544425 30408 leader_election.cc:302] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13640ac89a134ab6a7bcf832c19d75a9, 9be9f573e8694c37abfe05204bb7dee5; no voters:
I20260926 05:50:43.544584 30807 raft_consensus.cc:2781] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:43.544641 30807 raft_consensus.cc:482] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:43.544657 30807 raft_consensus.cc:3037] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:43.545538 30807 raft_consensus.cc:504] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.545697 30807 leader_election.cc:288] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [CANDIDATE]: Term 1 election: Requested vote from peers abe0d24fd75342e091717bca24c03b60 (127.11.247.2:37255), 9be9f573e8694c37abfe05204bb7dee5 (127.11.247.3:37025)
I20260926 05:50:43.545946 30740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f79633561b493c86bf28ebeef23bf7" candidate_uuid: "13640ac89a134ab6a7bcf832c19d75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be9f573e8694c37abfe05204bb7dee5"
I20260926 05:50:43.546044 30740 raft_consensus.cc:3037] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:43.546670 30606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f79633561b493c86bf28ebeef23bf7" candidate_uuid: "13640ac89a134ab6a7bcf832c19d75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe0d24fd75342e091717bca24c03b60"
I20260926 05:50:43.546764 30606 raft_consensus.cc:3037] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:43.547117 30740 raft_consensus.cc:2454] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13640ac89a134ab6a7bcf832c19d75a9 in term 1.
I20260926 05:50:43.547832 30606 raft_consensus.cc:2454] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13640ac89a134ab6a7bcf832c19d75a9 in term 1.
I20260926 05:50:43.548044 30408 leader_election.cc:302] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13640ac89a134ab6a7bcf832c19d75a9, 9be9f573e8694c37abfe05204bb7dee5; no voters:
I20260926 05:50:43.548334 30807 raft_consensus.cc:2781] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:43.548568 30807 raft_consensus.cc:686] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 1 LEADER]: Becoming Leader. State: Replica: 13640ac89a134ab6a7bcf832c19d75a9, State: Running, Role: LEADER
I20260926 05:50:43.548681 30807 consensus_queue.cc:237] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } }
I20260926 05:50:43.549602 30327 catalog_manager.cc:5495] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13640ac89a134ab6a7bcf832c19d75a9 (127.11.247.1). New cstate: current_term: 1 leader_uuid: "13640ac89a134ab6a7bcf832c19d75a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13640ac89a134ab6a7bcf832c19d75a9" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 39847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20260926 05:50:43.586218 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.583105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.11.247.1:39847)
0926 05:50:43.583177 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.583180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.583190 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:50:43.583270 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.583271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.583292 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.583387 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:43.583549 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.584209 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.585135 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.585449 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.585525 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.585743 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:43.585842 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:43.585920 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.586068 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.586115 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:50:43.588189 30740 raft_consensus.cc:1261] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 1 FOLLOWER]: Refusing update from remote peer 13640ac89a134ab6a7bcf832c19d75a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:43.588191 30606 raft_consensus.cc:1261] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 1 FOLLOWER]: Refusing update from remote peer 13640ac89a134ab6a7bcf832c19d75a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:43.588589 30807 consensus_queue.cc:1035] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be9f573e8694c37abfe05204bb7dee5" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:43.588766 30814 consensus_queue.cc:1035] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe0d24fd75342e091717bca24c03b60" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:43.591445 30819 mvcc.cc:205] Tried to move back new op lower bound from 7333485951334391808 to 7333485951176048640. Current Snapshot: MvccSnapshot[applied={T|T < 7333485951334391808}]
I20260926 05:50:43.592079 30818 mvcc.cc:205] Tried to move back new op lower bound from 7333485951334391808 to 7333485951176048640. Current Snapshot: MvccSnapshot[applied={T|T < 7333485951334391808 or (T in {7333485951334391808})}]
I20260926 05:50:43.594462 30327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:43.594584 30327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:43.595446 30327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56678ec2db0d48d59e19c1b2c662c0d0: Sending DeleteTablet for 3 replicas of tablet 61f79633561b493c86bf28ebeef23bf7
I20260926 05:50:43.595824 30586 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f79633561b493c86bf28ebeef23bf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:33712
I20260926 05:50:43.595830 30720 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f79633561b493c86bf28ebeef23bf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:49570
I20260926 05:50:43.596203 30452 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f79633561b493c86bf28ebeef23bf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:43 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:35644
I20260926 05:50:43.596431 30829 tablet_replica.cc:330] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9: stopping tablet replica
I20260926 05:50:43.596441 30828 tablet_replica.cc:330] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: stopping tablet replica
I20260926 05:50:43.596510 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30389
I20260926 05:50:43.596549 30828 raft_consensus.cc:2229] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:43.596571 30829 raft_consensus.cc:2229] T 61f79633561b493c86bf28ebeef23bf7 P 13640ac89a134ab6a7bcf832c19d75a9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:50:43.596684 30828 raft_consensus.cc:2258] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:43.598457 30827 tablet_replica.cc:330] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: stopping tablet replica
I20260926 05:50:43.598628 30828 ts_tablet_manager.cc:1905] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:43.598636 30827 raft_consensus.cc:2229] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:43.598794 30827 raft_consensus.cc:2258] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:43.599262 30827 ts_tablet_manager.cc:1905] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:43.600606 30828 ts_tablet_manager.cc:1918] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:43.600694 30828 log.cc:1196] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/61f79633561b493c86bf28ebeef23bf7
I20260926 05:50:43.600886 30827 ts_tablet_manager.cc:1918] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:43.600970 30828 ts_tablet_manager.cc:1939] T 61f79633561b493c86bf28ebeef23bf7 P 9be9f573e8694c37abfe05204bb7dee5: Deleting consensus metadata
I20260926 05:50:43.600970 30827 log.cc:1196] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/61f79633561b493c86bf28ebeef23bf7
I20260926 05:50:43.601238 30827 ts_tablet_manager.cc:1939] T 61f79633561b493c86bf28ebeef23bf7 P abe0d24fd75342e091717bca24c03b60: Deleting consensus metadata
I20260926 05:50:43.601444 30314 catalog_manager.cc:4841] TS 9be9f573e8694c37abfe05204bb7dee5 (127.11.247.3:37025): tablet 61f79633561b493c86bf28ebeef23bf7 (table test-table [id=e35b51d3a90b4fa69d87d4558871809e]) successfully deleted
I20260926 05:50:43.601656 30313 catalog_manager.cc:4841] TS abe0d24fd75342e091717bca24c03b60 (127.11.247.2:37255): tablet 61f79633561b493c86bf28ebeef23bf7 (table test-table [id=e35b51d3a90b4fa69d87d4558871809e]) successfully deleted
W20260926 05:50:43.602468 30311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:39847 (error 108)
I20260926 05:50:43.602753 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30523
I20260926 05:50:43.603665 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.602944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:39847 (local address 127.0.0.1:35676)
0926 05:50:43.603592 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.603634 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:39847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628}
W20260926 05:50:43.603809 30311 catalog_manager.cc:4584] TS 13640ac89a134ab6a7bcf832c19d75a9 (127.11.247.1:39847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61f79633561b493c86bf28ebeef23bf7: Network error: Client connection negotiation failed: client connection to 127.11.247.1:39847: connect: Connection refused (error 111)
I20260926 05:50:43.609028 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30657
I20260926 05:50:43.613865 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30295
2026-09-26T05:50:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2240 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T05:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:43Z Disabled control of system clock
I20260926 05:50:43.643173 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33873
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:33873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:43.725484 30834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:43.725721 30834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:43.725744 30834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:43.726926 30834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:43.726970 30834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:43.726984 30834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:43.726995 30834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:43.727008 30834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:43.728440 30834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:33873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:33873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:43.728674 30834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:43.728924 30834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:43.733726 30840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:43.733721 30842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:43.734019 30834 server_base.cc:1034] running on GCE node
W20260926 05:50:43.734038 30839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:43.734316 30834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:43.734699 30834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:43.735863 30834 hybrid_clock.cc:648] HybridClock initialized: now 1790401843735840 us; error 40 us; skew 500 ppm
I20260926 05:50:43.737210 30834 webserver.cc:460] Webserver started at http://127.11.247.62:41927/ using document root <none> and password file <none>
I20260926 05:50:43.737512 30834 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:43.737562 30834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:43.737658 30834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:43.738809 30834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "ca77f10073924bca96f74b9c2f9e648e"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "7b79ae54205731d50071be4a467591e6"
server_key_iv: "9a862e06ef9246576a91edd15b5794fb"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.739125 30834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "ca77f10073924bca96f74b9c2f9e648e"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "7b79ae54205731d50071be4a467591e6"
server_key_iv: "9a862e06ef9246576a91edd15b5794fb"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.740420 30834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:43.741180 30848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.741359 30834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:43.741469 30834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "ca77f10073924bca96f74b9c2f9e648e"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "7b79ae54205731d50071be4a467591e6"
server_key_iv: "9a862e06ef9246576a91edd15b5794fb"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.741564 30834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:43.756034 30834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:43.756409 30834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:43.756577 30834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:43.760807 30834 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:33873
I20260926 05:50:43.760845 30900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:33873 every 8 connection(s)
I20260926 05:50:43.761252 30834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:43.762233 30901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:43.764683 30901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: Bootstrap starting.
I20260926 05:50:43.765333 30901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:43.765673 30901 log.cc:824] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:43.766381 30901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: No bootstrap required, opened a new log
I20260926 05:50:43.767916 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30834
I20260926 05:50:43.768003 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:43.768215 12252 external_mini_cluster.cc:1428] Setting key 5153847e0a7d1bff2a5b94606c5fbbcc
I20260926 05:50:43.768168 30901 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca77f10073924bca96f74b9c2f9e648e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33873 } }
I20260926 05:50:43.768364 30901 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:43.768411 30901 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca77f10073924bca96f74b9c2f9e648e, State: Initialized, Role: FOLLOWER
I20260926 05:50:43.768606 30901 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca77f10073924bca96f74b9c2f9e648e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33873 } }
I20260926 05:50:43.768693 30901 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:43.768735 30901 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:43.768778 30901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:43.769487 30901 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca77f10073924bca96f74b9c2f9e648e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33873 } }
I20260926 05:50:43.769601 30901 leader_election.cc:302] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca77f10073924bca96f74b9c2f9e648e; no voters:
I20260926 05:50:43.769766 30901 leader_election.cc:288] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:43.769816 30906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:43.769977 30906 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [term 1 LEADER]: Becoming Leader. State: Replica: ca77f10073924bca96f74b9c2f9e648e, State: Running, Role: LEADER
I20260926 05:50:43.770031 30901 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:43.770087 30906 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca77f10073924bca96f74b9c2f9e648e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33873 } }
I20260926 05:50:43.770445 30906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca77f10073924bca96f74b9c2f9e648e. Latest consensus state: current_term: 1 leader_uuid: "ca77f10073924bca96f74b9c2f9e648e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca77f10073924bca96f74b9c2f9e648e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33873 } } }
I20260926 05:50:43.770525 30906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:43.770457 30907 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca77f10073924bca96f74b9c2f9e648e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca77f10073924bca96f74b9c2f9e648e" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 33873 } } }
I20260926 05:50:43.770568 30907 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e [sys.catalog]: This master's current role is: LEADER
W20260926 05:50:43.771991 30922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:50:43.772127 30922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:50:43.772181 30913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:43.772362 30913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:43.774319 30913 catalog_manager.cc:1347] Generated new cluster ID: 0ea256c85d234d5fa2fa02ec398a531e
I20260926 05:50:43.774371 30913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:43.774765 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.768735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.11.247.62:33873)
0926 05:50:43.769057 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.769065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.769088 (+ 23us) server_negotiation.cc:407] Connection header received
0926 05:50:43.769193 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.769199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.769262 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.769365 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:43.769651 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.771385 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.772525 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.773120 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.773239 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.773979 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:43.773982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:43.773982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:43.774145 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:43.774178 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:43.774189 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:43.774275 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.774399 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.774482 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260926 05:50:43.783183 30913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:43.783717 30913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:43.787245 30913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: Generated new TSK 0
I20260926 05:50:43.787461 30913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:43.789063 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33873
--builtin_ntp_servers=127.11.247.20:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:43.889247 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:43.889477 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:43.889503 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:43.890818 30925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:43.890861 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:43.890888 30925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:43.892307 30925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:43.892583 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:43.892915 30925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:43.899001 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:43.899019 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:43.899287 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:43.899451 30925 server_base.cc:1034] running on GCE node
I20260926 05:50:43.899673 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:43.899968 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:43.901124 30925 hybrid_clock.cc:648] HybridClock initialized: now 1790401843901102 us; error 39 us; skew 500 ppm
I20260926 05:50:43.902491 30925 webserver.cc:460] Webserver started at http://127.11.247.1:46661/ using document root <none> and password file <none>
I20260926 05:50:43.902797 30925 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:43.902872 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:43.903041 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:43.904167 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "3b983b9bbfa24b5aaa7379448548d45d"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "75b93a8333b8b20b97281e9e7067f2d3"
server_key_iv: "6bf47b45a03c81712c3d6175de2f615a"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.904572 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "3b983b9bbfa24b5aaa7379448548d45d"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "75b93a8333b8b20b97281e9e7067f2d3"
server_key_iv: "6bf47b45a03c81712c3d6175de2f615a"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.906066 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:43.906955 30939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.907145 30925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:43.907239 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "3b983b9bbfa24b5aaa7379448548d45d"
format_stamp: "Formatted at 2026-09-26 05:50:43 on dist-test-slave-21p4"
server_key: "75b93a8333b8b20b97281e9e7067f2d3"
server_key_iv: "6bf47b45a03c81712c3d6175de2f615a"
server_key_version: "encryptionkey@0"
I20260926 05:50:43.907338 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:43.916729 30925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:43.917097 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:43.917275 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:43.917532 30925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:43.917876 30925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:43.917932 30925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.917968 30925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:43.918015 30925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:43.923573 30925 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:35267
I20260926 05:50:43.923666 31052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:35267 every 8 connection(s)
I20260926 05:50:43.924016 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:43.924937 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 30925
I20260926 05:50:43.925060 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:50:43.925321 12252 external_mini_cluster.cc:1428] Setting key 5f9310a919929821bd0234b45a4dd8f9
I20260926 05:50:43.926239 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33873
--builtin_ntp_servers=127.11.247.20:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:50:43.930969 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.924804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:54605 (local address 127.11.247.62:33873)
0926 05:50:43.924850 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:43.924852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:43.925155 (+ 303us) server_negotiation.cc:407] Connection header received
0926 05:50:43.925940 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:43.925941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:43.925953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:43.925983 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:43.927050 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.927804 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.929003 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.929338 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.929382 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.930570 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:43.930572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:43.930572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:43.930608 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:43.930625 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:43.930630 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:43.930664 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:43.930814 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 05:50:43.930852 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:50:43.931547 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:43.924804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33873 (local address 127.11.247.1:54605)
0926 05:50:43.925001 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:43.925046 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:43.925813 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:43.926158 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:43.926162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:43.926172 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:43.926938 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.926945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.927923 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.927924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.928890 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:43.928894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:43.929492 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:43.929493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:43.929663 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:43.930366 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:43.930367 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:43.930411 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:43.930457 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:43.930712 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:43.930719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:43.930722 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:43.931090 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 05:50:43.931229 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260926 05:50:43.932268 31053 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33873
I20260926 05:50:43.932384 31053 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:43.932587 31053 heartbeater.cc:511] Master 127.11.247.62:33873 requested a full tablet report, sending...
I20260926 05:50:43.933192 30865 ts_manager.cc:194] Registered new tserver with Master: 3b983b9bbfa24b5aaa7379448548d45d (127.11.247.1:35267)
I20260926 05:50:43.934234 30865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.1:54605
W20260926 05:50:44.012877 31056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:44.013146 31056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:44.013175 31056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:44.014876 31056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:44.014959 31056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:44.015000 31056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:44.017103 31056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:44.017380 31056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:44.017664 31056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:44.022987 31062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:44.022979 31061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:44.022979 31064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:44.023373 31056 server_base.cc:1034] running on GCE node
I20260926 05:50:44.023669 31056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:44.023980 31056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:44.025152 31056 hybrid_clock.cc:648] HybridClock initialized: now 1790401844025133 us; error 43 us; skew 500 ppm
I20260926 05:50:44.026636 31056 webserver.cc:460] Webserver started at http://127.11.247.2:38391/ using document root <none> and password file <none>
I20260926 05:50:44.026984 31056 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:44.027068 31056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:44.027238 31056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:44.028419 31056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "326a79a09351418db796746382767444"
format_stamp: "Formatted at 2026-09-26 05:50:44 on dist-test-slave-21p4"
server_key: "f0372d08d14e1574b6829cf78e806b18"
server_key_iv: "9f82e44b0620ff43f7bd964252f6c981"
server_key_version: "encryptionkey@0"
I20260926 05:50:44.028833 31056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "326a79a09351418db796746382767444"
format_stamp: "Formatted at 2026-09-26 05:50:44 on dist-test-slave-21p4"
server_key: "f0372d08d14e1574b6829cf78e806b18"
server_key_iv: "9f82e44b0620ff43f7bd964252f6c981"
server_key_version: "encryptionkey@0"
I20260926 05:50:44.030283 31056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:44.031339 31070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:44.031591 31056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:44.031705 31056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "326a79a09351418db796746382767444"
format_stamp: "Formatted at 2026-09-26 05:50:44 on dist-test-slave-21p4"
server_key: "f0372d08d14e1574b6829cf78e806b18"
server_key_iv: "9f82e44b0620ff43f7bd964252f6c981"
server_key_version: "encryptionkey@0"
I20260926 05:50:44.031849 31056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:44.046234 31056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:44.046542 31056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:44.046749 31056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:44.046963 31056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:44.047302 31056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:44.047335 31056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:44.047391 31056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:44.047418 31056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:44.053388 31056 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:37929
I20260926 05:50:44.053519 31183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:37929 every 8 connection(s)
I20260926 05:50:44.053855 31056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:50:44.059846 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.054703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:50535 (local address 127.11.247.62:33873)
0926 05:50:44.054763 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.054765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.055094 (+ 329us) server_negotiation.cc:407] Connection header received
0926 05:50:44.055668 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.055670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.055682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.055740 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:44.056515 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.057040 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.058120 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.058339 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.058427 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.059495 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:44.059496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:44.059497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:44.059504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:44.059528 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:44.059533 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:44.059599 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.059705 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.059746 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:50:44.060101 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.054699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33873 (local address 127.11.247.2:50535)
0926 05:50:44.054947 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.054990 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.055554 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.055823 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.055826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.055835 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:44.056382 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.056388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.057153 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.057154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.058006 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.058009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.058451 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.058452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.058639 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.059301 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:44.059301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:44.059335 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:44.059364 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:44.059601 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:44.059608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:44.059612 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.059707 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.059837 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":130,"threads_started":1}
I20260926 05:50:44.061030 31184 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33873
I20260926 05:50:44.061146 31184 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:44.061394 31184 heartbeater.cc:511] Master 127.11.247.62:33873 requested a full tablet report, sending...
I20260926 05:50:44.061972 30865 ts_manager.cc:194] Registered new tserver with Master: 326a79a09351418db796746382767444 (127.11.247.2:37929)
I20260926 05:50:44.062438 30865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.2:50535
I20260926 05:50:44.062788 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31056
I20260926 05:50:44.062908 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:50:44.063140 12252 external_mini_cluster.cc:1428] Setting key da1d0722fb643f5e9ca8b6dda4aa4132
I20260926 05:50:44.064231 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33873
--builtin_ntp_servers=127.11.247.20:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:44.148226 31187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:44.148450 31187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:44.148494 31187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:44.149852 31187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:44.149904 31187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:44.149942 31187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:44.151468 31187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:33873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:44.151778 31187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:44.152139 31187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:44.157789 31193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:44.157814 31192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:44.157789 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:44.158191 31187 server_base.cc:1034] running on GCE node
I20260926 05:50:44.158406 31187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:44.158720 31187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:44.159879 31187 hybrid_clock.cc:648] HybridClock initialized: now 1790401844159866 us; error 32 us; skew 500 ppm
I20260926 05:50:44.161253 31187 webserver.cc:460] Webserver started at http://127.11.247.3:34131/ using document root <none> and password file <none>
I20260926 05:50:44.161497 31187 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:44.161544 31187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:44.161644 31187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:44.162802 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "2d2ebf7cea76464bba45812ba921cc81"
format_stamp: "Formatted at 2026-09-26 05:50:44 on dist-test-slave-21p4"
server_key: "5881475ca0dee2b1e6274431c28fdb02"
server_key_iv: "d63ac8f21635621e4bd8dd2d73d9168e"
server_key_version: "encryptionkey@0"
I20260926 05:50:44.163141 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "2d2ebf7cea76464bba45812ba921cc81"
format_stamp: "Formatted at 2026-09-26 05:50:44 on dist-test-slave-21p4"
server_key: "5881475ca0dee2b1e6274431c28fdb02"
server_key_iv: "d63ac8f21635621e4bd8dd2d73d9168e"
server_key_version: "encryptionkey@0"
I20260926 05:50:44.164505 31187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:44.165395 31201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:44.165624 31187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:44.165726 31187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "2d2ebf7cea76464bba45812ba921cc81"
format_stamp: "Formatted at 2026-09-26 05:50:44 on dist-test-slave-21p4"
server_key: "5881475ca0dee2b1e6274431c28fdb02"
server_key_iv: "d63ac8f21635621e4bd8dd2d73d9168e"
server_key_version: "encryptionkey@0"
I20260926 05:50:44.165834 31187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:44.179728 31187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:44.180044 31187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:44.180151 31187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:44.180354 31187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:44.180646 31187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:44.180677 31187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:44.180703 31187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:44.180716 31187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:44.186380 31187 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:36205
I20260926 05:50:44.186467 31314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:36205 every 8 connection(s)
I20260926 05:50:44.186863 31187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:50:44.189683 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31187
I20260926 05:50:44.189803 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:50:44.190027 12252 external_mini_cluster.cc:1428] Setting key 72ab6d768af4c89bcc0d6e1be8a5f128
I20260926 05:50:44.192890 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.187715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:51515 (local address 127.11.247.62:33873)
0926 05:50:44.187793 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.187795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.188039 (+ 244us) server_negotiation.cc:407] Connection header received
0926 05:50:44.188648 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.188649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.188660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.188714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:44.189393 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.190017 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.191479 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.191670 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.191729 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.192614 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:44.192615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:44.192615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:44.192620 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:44.192631 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:44.192635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:44.192677 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.192805 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.192830 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:50:44.193193 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.187703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:33873 (local address 127.11.247.3:51515)
0926 05:50:44.187910 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.187946 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.188550 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.188777 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.188780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.188789 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:44.189288 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.189319 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.190145 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.190146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.191405 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.191409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.191745 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.191746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.191901 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.192487 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:44.192487 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:44.192517 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:44.192542 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:44.192715 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:44.192722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:44.192725 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.192835 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.192927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260926 05:50:44.193677 31315 heartbeater.cc:347] Connected to a master server at 127.11.247.62:33873
I20260926 05:50:44.193786 31315 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:44.194024 31315 heartbeater.cc:511] Master 127.11.247.62:33873 requested a full tablet report, sending...
I20260926 05:50:44.194440 30865 ts_manager.cc:194] Registered new tserver with Master: 2d2ebf7cea76464bba45812ba921cc81 (127.11.247.3:36205)
I20260926 05:50:44.194887 30865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.3:51515
I20260926 05:50:44.202208 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:44.206817 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.203723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.11.247.62:33873)
0926 05:50:44.203803 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.203805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.203969 (+ 164us) server_negotiation.cc:407] Connection header received
0926 05:50:44.204012 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.204013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.204026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.204067 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:44.204209 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.204694 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.205470 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.205653 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.205705 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.206430 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:44.206431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:44.206431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:44.206436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:44.206452 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:44.206456 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:44.206531 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.206690 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.206705 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:50:44.208631 30865 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:44.210512 30865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:44.217049 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.212913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:36205 (local address 127.0.0.1:57162)
0926 05:50:44.213095 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.213106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.213163 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.213501 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.213504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.213506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:44.213579 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.213581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.214698 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.214699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.215905 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.215908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.216741 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.216742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.216820 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.216833 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.216899 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.216966 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260926 05:50:44.217060 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.213017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.11.247.3:36205)
0926 05:50:44.213307 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.213312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.213324 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:50:44.213365 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.213367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.213382 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.213479 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:44.213667 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.214555 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.216007 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.216648 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.216738 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.216838 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.216926 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.216966 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":83,"threads_started":1}
I20260926 05:50:44.217401 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.212868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35267 (local address 127.0.0.1:42202)
0926 05:50:44.213035 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.213079 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.213161 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.214015 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.214015 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.214018 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:44.214062 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.214063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.215353 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.215354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.216243 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.216245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.217157 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.217158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.217225 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.217230 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.217261 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.217319 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":111,"threads_started":1}
I20260926 05:50:44.217517 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.213132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:37929 (local address 127.0.0.1:34262)
0926 05:50:44.213379 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.213389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.213439 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.213728 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.213729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.213731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:44.213789 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.213791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.215178 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.215179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.216224 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.216227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.217242 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.217242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.217295 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.217297 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.217349 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.217383 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":63,"threads_started":1}
I20260926 05:50:44.217523 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.213355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.11.247.2:37929)
0926 05:50:44.213549 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.213555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.213570 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:50:44.213602 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.213603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.213615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.213699 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:44.213875 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.215081 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.216316 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.217051 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.217126 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.217211 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.217390 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.217435 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260926 05:50:44.217813 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.213209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.11.247.1:35267)
0926 05:50:44.213845 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.213849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.213860 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:50:44.213898 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.213900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.213916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.214131 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:44.214193 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.215265 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.216357 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.217056 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.217622 (+ 566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.217692 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.217713 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.217746 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":85,"threads_started":1}
I20260926 05:50:44.218405 31118 tablet_service.cc:1467] Processing CreateTablet for tablet b3b3e4b5604b4d9da3e49b8008d52c11 (DEFAULT_TABLE table=test-table [id=3553e36f29e84fa98102057fa717bb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:44.218504 30987 tablet_service.cc:1467] Processing CreateTablet for tablet b3b3e4b5604b4d9da3e49b8008d52c11 (DEFAULT_TABLE table=test-table [id=3553e36f29e84fa98102057fa717bb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:44.218842 31118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b3e4b5604b4d9da3e49b8008d52c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:44.218869 30987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b3e4b5604b4d9da3e49b8008d52c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:44.218786 31249 tablet_service.cc:1467] Processing CreateTablet for tablet b3b3e4b5604b4d9da3e49b8008d52c11 (DEFAULT_TABLE table=test-table [id=3553e36f29e84fa98102057fa717bb08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:44.219110 31249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b3e4b5604b4d9da3e49b8008d52c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:44.221519 31329 tablet_bootstrap.cc:492] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d: Bootstrap starting.
I20260926 05:50:44.222316 31329 tablet_bootstrap.cc:654] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:44.222371 31331 tablet_bootstrap.cc:492] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Bootstrap starting.
I20260926 05:50:44.222736 31329 log.cc:824] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:44.223078 31331 tablet_bootstrap.cc:654] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:44.223385 31331 log.cc:824] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:44.223489 31329 tablet_bootstrap.cc:492] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d: No bootstrap required, opened a new log
I20260926 05:50:44.223582 31329 ts_tablet_manager.cc:1397] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:50:44.223750 31330 tablet_bootstrap.cc:492] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Bootstrap starting.
I20260926 05:50:44.224261 31330 tablet_bootstrap.cc:654] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:44.224561 31330 log.cc:824] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:44.225121 31331 tablet_bootstrap.cc:492] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: No bootstrap required, opened a new log
I20260926 05:50:44.225210 31331 ts_tablet_manager.cc:1397] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 05:50:44.225327 31330 tablet_bootstrap.cc:492] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: No bootstrap required, opened a new log
I20260926 05:50:44.225386 31330 ts_tablet_manager.cc:1397] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:50:44.226025 31329 raft_consensus.cc:348] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.226243 31329 raft_consensus.cc:374] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:44.226306 31329 raft_consensus.cc:729] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b983b9bbfa24b5aaa7379448548d45d, State: Initialized, Role: FOLLOWER
I20260926 05:50:44.226467 31329 consensus_queue.cc:260] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.226804 31329 ts_tablet_manager.cc:1428] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:50:44.226966 31053 heartbeater.cc:503] Master 127.11.247.62:33873 was elected leader, sending a full tablet report...
I20260926 05:50:44.228051 31330 raft_consensus.cc:348] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.228232 31330 raft_consensus.cc:374] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:44.228276 31330 raft_consensus.cc:729] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2ebf7cea76464bba45812ba921cc81, State: Initialized, Role: FOLLOWER
I20260926 05:50:44.228175 31331 raft_consensus.cc:348] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.228349 31331 raft_consensus.cc:374] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:44.228400 31331 raft_consensus.cc:729] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326a79a09351418db796746382767444, State: Initialized, Role: FOLLOWER
I20260926 05:50:44.228396 31330 consensus_queue.cc:260] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.228529 31331 consensus_queue.cc:260] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.228696 31315 heartbeater.cc:503] Master 127.11.247.62:33873 was elected leader, sending a full tablet report...
I20260926 05:50:44.228679 31330 ts_tablet_manager.cc:1428] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:50:44.228772 31331 ts_tablet_manager.cc:1428] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 05:50:44.228909 31184 heartbeater.cc:503] Master 127.11.247.62:33873 was elected leader, sending a full tablet report...
I20260926 05:50:44.235610 31335 raft_consensus.cc:482] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:44.235735 31335 raft_consensus.cc:504] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.236192 31335 leader_election.cc:288] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d2ebf7cea76464bba45812ba921cc81 (127.11.247.3:36205), 326a79a09351418db796746382767444 (127.11.247.2:37929)
I20260926 05:50:44.239341 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.236438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:36205 (local address 127.11.247.1:45755)
0926 05:50:44.236498 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.236511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.236569 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.236737 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.236738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.236739 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:44.236814 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.236816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.237506 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.237507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.238274 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.238276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.239086 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.239087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.239145 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.239149 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.239192 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.239237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:50:44.239341 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.236553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:45755 (local address 127.11.247.3:36205)
0926 05:50:44.236613 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.236616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.236623 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:50:44.236659 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.236660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.236668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.236733 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:44.236921 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.237416 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.238371 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.239000 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.239082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.239091 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.239217 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.239265 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:50:44.239589 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.236806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:44917 (local address 127.11.247.2:37929)
0926 05:50:44.236882 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.236884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.236894 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:50:44.236975 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.236976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.236992 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.237053 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:44.237209 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.237674 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.238660 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.239224 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.239288 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.239295 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.239472 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.239526 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:50:44.239709 31269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b3e4b5604b4d9da3e49b8008d52c11" candidate_uuid: "3b983b9bbfa24b5aaa7379448548d45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2ebf7cea76464bba45812ba921cc81" is_pre_election: true
I20260926 05:50:44.239817 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.236550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:37929 (local address 127.11.247.1:44917)
0926 05:50:44.236818 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.236848 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:44.236908 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:44.237057 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:44.237057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:44.237059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:44.237133 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.237140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.237761 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.237762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.238506 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:44.238508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.239309 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.239309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:44.239374 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.239378 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:44.239673 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 05:50:44.239742 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":129,"threads_started":1}
I20260926 05:50:44.239881 31269 raft_consensus.cc:2454] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b983b9bbfa24b5aaa7379448548d45d in term 0.
I20260926 05:50:44.240101 31138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b3e4b5604b4d9da3e49b8008d52c11" candidate_uuid: "3b983b9bbfa24b5aaa7379448548d45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326a79a09351418db796746382767444" is_pre_election: true
I20260926 05:50:44.240171 30940 leader_election.cc:302] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2ebf7cea76464bba45812ba921cc81, 3b983b9bbfa24b5aaa7379448548d45d; no voters:
I20260926 05:50:44.240209 31138 raft_consensus.cc:2454] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b983b9bbfa24b5aaa7379448548d45d in term 0.
I20260926 05:50:44.240320 31335 raft_consensus.cc:2781] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:44.240391 31335 raft_consensus.cc:482] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:44.240415 31335 raft_consensus.cc:3037] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:44.241146 31335 raft_consensus.cc:504] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.241289 31335 leader_election.cc:288] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [CANDIDATE]: Term 1 election: Requested vote from peers 2d2ebf7cea76464bba45812ba921cc81 (127.11.247.3:36205), 326a79a09351418db796746382767444 (127.11.247.2:37929)
I20260926 05:50:44.241484 31138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b3e4b5604b4d9da3e49b8008d52c11" candidate_uuid: "3b983b9bbfa24b5aaa7379448548d45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326a79a09351418db796746382767444"
I20260926 05:50:44.241494 31269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b3e4b5604b4d9da3e49b8008d52c11" candidate_uuid: "3b983b9bbfa24b5aaa7379448548d45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2ebf7cea76464bba45812ba921cc81"
I20260926 05:50:44.241593 31269 raft_consensus.cc:3037] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:44.241570 31138 raft_consensus.cc:3037] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:44.242352 31269 raft_consensus.cc:2454] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b983b9bbfa24b5aaa7379448548d45d in term 1.
I20260926 05:50:44.242349 31138 raft_consensus.cc:2454] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b983b9bbfa24b5aaa7379448548d45d in term 1.
I20260926 05:50:44.242558 30940 leader_election.cc:302] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326a79a09351418db796746382767444, 3b983b9bbfa24b5aaa7379448548d45d; no voters:
I20260926 05:50:44.242707 31335 raft_consensus.cc:2781] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:44.242892 31335 raft_consensus.cc:686] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [term 1 LEADER]: Becoming Leader. State: Replica: 3b983b9bbfa24b5aaa7379448548d45d, State: Running, Role: LEADER
I20260926 05:50:44.243026 31335 consensus_queue.cc:237] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } }
I20260926 05:50:44.243918 30865 catalog_manager.cc:5495] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b983b9bbfa24b5aaa7379448548d45d (127.11.247.1). New cstate: current_term: 1 leader_uuid: "3b983b9bbfa24b5aaa7379448548d45d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b983b9bbfa24b5aaa7379448548d45d" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 } health_report { overall_health: UNKNOWN } } }
I20260926 05:50:44.249586 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.247073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.11.247.1:35267)
0926 05:50:44.247118 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:44.247120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:44.247129 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:44.247205 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:44.247206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:44.247225 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:44.247295 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:44.247390 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.247939 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.248766 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:44.248996 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:44.249078 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:44.249206 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:44.249296 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:44.249370 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:44.249423 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:50:44.249433 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:50:44.251564 31138 raft_consensus.cc:1261] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 1 FOLLOWER]: Refusing update from remote peer 3b983b9bbfa24b5aaa7379448548d45d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:44.251564 31269 raft_consensus.cc:1261] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 1 FOLLOWER]: Refusing update from remote peer 3b983b9bbfa24b5aaa7379448548d45d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:44.251917 31335 consensus_queue.cc:1035] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d2ebf7cea76464bba45812ba921cc81" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:44.252115 31339 consensus_queue.cc:1035] T b3b3e4b5604b4d9da3e49b8008d52c11 P 3b983b9bbfa24b5aaa7379448548d45d [LEADER]: Connected to new peer: Peer: permanent_uuid: "326a79a09351418db796746382767444" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:44.254701 31342 mvcc.cc:205] Tried to move back new op lower bound from 7333485954051428352 to 7333485954020151296. Current Snapshot: MvccSnapshot[applied={T|T < 7333485954051428352}]
I20260926 05:50:44.255486 31344 mvcc.cc:205] Tried to move back new op lower bound from 7333485954051428352 to 7333485954020151296. Current Snapshot: MvccSnapshot[applied={T|T < 7333485954051428352 or (T in {7333485954051428352})}]
I20260926 05:50:44.257659 30865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:44.257759 30865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:44.258685 30865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca77f10073924bca96f74b9c2f9e648e: Sending DeleteTablet for 3 replicas of tablet b3b3e4b5604b4d9da3e49b8008d52c11
I20260926 05:50:44.259047 30987 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b3e4b5604b4d9da3e49b8008d52c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:44 UTC) from {username='slave'} at 127.0.0.1:42202
I20260926 05:50:44.259095 31249 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b3e4b5604b4d9da3e49b8008d52c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:44 UTC) from {username='slave'} at 127.0.0.1:57162
I20260926 05:50:44.259317 31118 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b3e4b5604b4d9da3e49b8008d52c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:44 UTC) from {username='slave'} at 127.0.0.1:34262
I20260926 05:50:44.259537 31354 tablet_replica.cc:330] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: stopping tablet replica
I20260926 05:50:44.259654 31354 raft_consensus.cc:2229] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:44.259819 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30925
I20260926 05:50:44.259775 31354 raft_consensus.cc:2258] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:44.260658 31354 ts_tablet_manager.cc:1905] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:44.262859 31354 ts_tablet_manager.cc:1918] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:44.262946 31354 log.cc:1196] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/b3b3e4b5604b4d9da3e49b8008d52c11
I20260926 05:50:44.263200 31354 ts_tablet_manager.cc:1939] T b3b3e4b5604b4d9da3e49b8008d52c11 P 326a79a09351418db796746382767444: Deleting consensus metadata
I20260926 05:50:44.263216 31353 tablet_replica.cc:330] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: stopping tablet replica
I20260926 05:50:44.263370 31353 raft_consensus.cc:2229] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:44.263509 31353 raft_consensus.cc:2258] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:44.263684 30849 catalog_manager.cc:4841] TS 326a79a09351418db796746382767444 (127.11.247.2:37929): tablet b3b3e4b5604b4d9da3e49b8008d52c11 (table test-table [id=3553e36f29e84fa98102057fa717bb08]) successfully deleted
I20260926 05:50:44.264019 31353 ts_tablet_manager.cc:1905] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:44.265986 31353 ts_tablet_manager.cc:1918] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:44.266081 31353 log.cc:1196] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/b3b3e4b5604b4d9da3e49b8008d52c11
I20260926 05:50:44.266345 31353 ts_tablet_manager.cc:1939] T b3b3e4b5604b4d9da3e49b8008d52c11 P 2d2ebf7cea76464bba45812ba921cc81: Deleting consensus metadata
I20260926 05:50:44.267172 30849 catalog_manager.cc:4841] TS 2d2ebf7cea76464bba45812ba921cc81 (127.11.247.3:36205): tablet b3b3e4b5604b4d9da3e49b8008d52c11 (table test-table [id=3553e36f29e84fa98102057fa717bb08]) successfully deleted
W20260926 05:50:44.268113 30850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:35267 (error 108)
I20260926 05:50:44.268790 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:44.268575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:35267 (local address 127.0.0.1:42224)
0926 05:50:44.268693 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:44.268751 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 05:50:44.268780 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31056
W20260926 05:50:44.268980 30850 catalog_manager.cc:4584] TS 3b983b9bbfa24b5aaa7379448548d45d (127.11.247.1:35267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3b3e4b5604b4d9da3e49b8008d52c11: Network error: Client connection negotiation failed: client connection to 127.11.247.1:35267: connect: Connection refused (error 111)
I20260926 05:50:44.274712 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31187
I20260926 05:50:44.279698 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 30834
2026-09-26T05:50:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (664 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:44 dist-test-slave-21p4 krb5kdc[31358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:44 dist-test-slave-21p4 krb5kdc[31358](info): set up 2 sockets
Sep 26 05:50:44 dist-test-slave-21p4 krb5kdc[31358](info): commencing operation
krb5kdc: starting...
W20260926 05:50:45.322461 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:50:45.445999 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37109
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:45.529798 31371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:45.530031 31371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:45.530056 31371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:45.531332 31371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:45.531386 31371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:45.531402 31371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:45.531414 31371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:45.531426 31371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:45.532912 31371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:45.533121 31371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:45.533304 31371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:45.538309 31379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:45.538312 31377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:45.538309 31376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:45.538365 31371 server_base.cc:1034] running on GCE node
I20260926 05:50:45.538951 31371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:45.539291 31371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:45.540478 31371 hybrid_clock.cc:648] HybridClock initialized: now 1790401845540456 us; error 36 us; skew 500 ppm
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:45.543196 31371 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:50:45.543732 31371 webserver.cc:460] Webserver started at http://127.11.247.62:39189/ using document root <none> and password file <none>
I20260926 05:50:45.543991 31371 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:45.544063 31371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:45.544214 31371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:45.545300 31371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "b61259e5b1c34996b644452e0800a899"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "5750737e0b94a538bc284482e729b427"
server_key_iv: "72edad062c201718d138b9b98e4eb76a"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.545656 31371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "b61259e5b1c34996b644452e0800a899"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "5750737e0b94a538bc284482e729b427"
server_key_iv: "72edad062c201718d138b9b98e4eb76a"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.547125 31371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:50:45.547945 31386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.548171 31371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:45.548278 31371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "b61259e5b1c34996b644452e0800a899"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "5750737e0b94a538bc284482e729b427"
server_key_iv: "72edad062c201718d138b9b98e4eb76a"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.548360 31371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:45.582047 31371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:45.584560 31371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:45.584743 31371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:45.589250 31371 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:37109
I20260926 05:50:45.589321 31438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:37109 every 8 connection(s)
I20260926 05:50:45.589704 31371 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:45.590934 31439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:45.591495 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31371
I20260926 05:50:45.591591 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:45.591817 12252 external_mini_cluster.cc:1428] Setting key 7d7a595421be8f1296026ea8cd039e0d
I20260926 05:50:45.594242 31439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899: Bootstrap starting.
I20260926 05:50:45.595011 31439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:45.595368 31439 log.cc:824] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:45.596251 31439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899: No bootstrap required, opened a new log
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:45.599130 31439 raft_consensus.cc:348] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61259e5b1c34996b644452e0800a899" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37109 } }
I20260926 05:50:45.599337 31439 raft_consensus.cc:374] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:45.599388 31439 raft_consensus.cc:729] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61259e5b1c34996b644452e0800a899, State: Initialized, Role: FOLLOWER
I20260926 05:50:45.599551 31439 consensus_queue.cc:260] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61259e5b1c34996b644452e0800a899" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37109 } }
I20260926 05:50:45.599653 31439 raft_consensus.cc:388] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:45.599694 31439 raft_consensus.cc:482] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:45.599737 31439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:45.600365 31439 raft_consensus.cc:504] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61259e5b1c34996b644452e0800a899" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37109 } }
I20260926 05:50:45.600526 31439 leader_election.cc:302] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b61259e5b1c34996b644452e0800a899; no voters:
I20260926 05:50:45.600708 31439 leader_election.cc:288] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:45.600754 31444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:45.601019 31439 sys_catalog.cc:564] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:45.601656 31444 raft_consensus.cc:686] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [term 1 LEADER]: Becoming Leader. State: Replica: b61259e5b1c34996b644452e0800a899, State: Running, Role: LEADER
I20260926 05:50:45.601775 31444 consensus_queue.cc:237] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61259e5b1c34996b644452e0800a899" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37109 } }
I20260926 05:50:45.602151 31444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b61259e5b1c34996b644452e0800a899. Latest consensus state: current_term: 1 leader_uuid: "b61259e5b1c34996b644452e0800a899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61259e5b1c34996b644452e0800a899" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37109 } } }
I20260926 05:50:45.602228 31444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:45.602368 31445 sys_catalog.cc:455] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b61259e5b1c34996b644452e0800a899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61259e5b1c34996b644452e0800a899" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37109 } } }
I20260926 05:50:45.602451 31445 sys_catalog.cc:458] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:45.602483 31458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:45.603061 31458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:45.603924 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:45.592407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.11.247.62:37109)
0926 05:50:45.592575 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:45.592583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:45.592611 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:50:45.593069 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:45.593076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:45.593125 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:45.593248 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:45.594102 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.595135 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.596208 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.596689 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.596776 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:45.599759 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:45.599764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:45.599764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:45.599777 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:45.601733 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:45.602210 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:45.602213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:45.602213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:45.602323 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:45.602670 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:45.602671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:45.602672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:45.602978 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:45.603161 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:45.603563 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 05:50:45.603712 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260926 05:50:45.605111 31458 catalog_manager.cc:1347] Generated new cluster ID: a904f290ddbb4e55aef03b237eeea134
I20260926 05:50:45.605165 31458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:45.616904 31458 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:45.617697 31458 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:45.622023 31458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899: Generated new TSK 0
I20260926 05:50:45.622274 31458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.247.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
Entry for principal kudu/127.11.247.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab.
I20260926 05:50:45.654899 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37109
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:45.740796 31465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:45.741065 31465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:45.741112 31465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:45.742419 31465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:45.742496 31465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:45.742547 31465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:45.744216 31465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.1.keytab
--principal=kudu/127.11.247.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:45.744520 31465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:45.744796 31465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:45.750185 31473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:45.750155 31470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:45.750171 31471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:45.750461 31465 server_base.cc:1034] running on GCE node
I20260926 05:50:45.750773 31465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:45.751065 31465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:45.752242 31465 hybrid_clock.cc:648] HybridClock initialized: now 1790401845752217 us; error 55 us; skew 500 ppm
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:45.755043 31465 init.cc:377] Logged in from keytab as kudu/127.11.247.1@KRBTEST.COM (short username kudu)
I20260926 05:50:45.755620 31465 webserver.cc:460] Webserver started at http://127.11.247.1:44775/ using document root <none> and password file <none>
I20260926 05:50:45.755887 31465 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:45.755961 31465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:45.756134 31465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:45.757254 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "b7c66faaef564a2ca3bbc2f3136210fe"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "9c1465b336c607e9c8431dee3ba883c9"
server_key_iv: "c00926f5c39ceb5d6ce859e30f6097a4"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.757653 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "b7c66faaef564a2ca3bbc2f3136210fe"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "9c1465b336c607e9c8431dee3ba883c9"
server_key_iv: "c00926f5c39ceb5d6ce859e30f6097a4"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.759258 31465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:45.760200 31480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.760473 31465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:45.760602 31465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "b7c66faaef564a2ca3bbc2f3136210fe"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "9c1465b336c607e9c8431dee3ba883c9"
server_key_iv: "c00926f5c39ceb5d6ce859e30f6097a4"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.760699 31465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:45.779599 31465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:45.781864 31465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:45.782043 31465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:45.782294 31465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:45.782678 31465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:45.782733 31465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.782774 31465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:45.782801 31465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.789108 31465 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:41117
I20260926 05:50:45.789227 31593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:41117 every 8 connection(s)
I20260926 05:50:45.789556 31465 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:45.790658 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31465
I20260926 05:50:45.790773 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:50:45.790990 12252 external_mini_cluster.cc:1428] Setting key b63e4f991cec2dc3e26937c41182a9e3
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.1@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.2@KRBTEST.COM; defaulting to no policy
I20260926 05:50:45.799942 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:45.790383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:57913 (local address 127.11.247.62:37109)
0926 05:50:45.790468 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:45.790470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:45.790774 (+ 304us) server_negotiation.cc:407] Connection header received
0926 05:50:45.791431 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:45.791432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:45.791443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:45.791489 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:45.792480 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.792992 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.794008 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.794240 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.794319 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:45.796844 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:45.796846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:45.796847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:45.796855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:45.798323 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:45.798875 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:45.798876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:45.798877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:45.798942 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:45.799215 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:45.799216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:45.799216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:45.799338 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:45.799424 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:45.799778 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 05:50:45.799872 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 05:50:45.800199 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:45.790351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37109 (local address 127.11.247.1:57913)
0926 05:50:45.790631 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:45.790685 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:45.791341 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:45.791560 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:45.791565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:45.792178 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:45.792378 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:45.792383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.793105 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.793106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:45.793917 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:45.793919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.794339 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.794340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:45.794474 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:45.795065 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:45.795073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:45.796741 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:45.798452 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:45.798459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:45.798462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:45.798797 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:45.799010 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:45.799011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:45.799011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:45.799139 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:45.799406 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:45.799414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:45.799568 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 05:50:45.799722 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 05:50:45.799906 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":141,"threads_started":1}
I20260926 05:50:45.800799 31594 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37109
I20260926 05:50:45.800925 31594 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:45.801169 31594 heartbeater.cc:511] Master 127.11.247.62:37109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.2@KRBTEST.COM" created.
I20260926 05:50:45.801718 31403 ts_manager.cc:194] Registered new tserver with Master: b7c66faaef564a2ca3bbc2f3136210fe (127.11.247.1:41117)
I20260926 05:50:45.802480 31403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.1@KRBTEST.COM'} at 127.11.247.1:57913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
Entry for principal kudu/127.11.247.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab.
I20260926 05:50:45.817982 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37109
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:45.917100 31599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:45.917348 31599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:45.917387 31599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:45.918550 31599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:45.918645 31599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:45.918694 31599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:45.920177 31599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.2.keytab
--principal=kudu/127.11.247.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:45.920439 31599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:45.920671 31599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:45.925716 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:45.925807 31607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:45.925716 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:45.925918 31599 server_base.cc:1034] running on GCE node
I20260926 05:50:45.926144 31599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:45.926397 31599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:45.927561 31599 hybrid_clock.cc:648] HybridClock initialized: now 1790401845927530 us; error 41 us; skew 500 ppm
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:45.936445 31599 init.cc:377] Logged in from keytab as kudu/127.11.247.2@KRBTEST.COM (short username kudu)
I20260926 05:50:45.936934 31599 webserver.cc:460] Webserver started at http://127.11.247.2:39561/ using document root <none> and password file <none>
I20260926 05:50:45.937177 31599 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:45.937248 31599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:45.937409 31599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:45.938539 31599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "b3343ebecd744ccfba349ef291ec0ade"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "8068a4890c813e94b1eb7b01010a77ad"
server_key_iv: "ca707171b6a0b70e6bf98a4cfbd8af2d"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.938977 31599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "b3343ebecd744ccfba349ef291ec0ade"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "8068a4890c813e94b1eb7b01010a77ad"
server_key_iv: "ca707171b6a0b70e6bf98a4cfbd8af2d"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.940353 31599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:45.941118 31614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.941296 31599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:50:45.941407 31599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "b3343ebecd744ccfba349ef291ec0ade"
format_stamp: "Formatted at 2026-09-26 05:50:45 on dist-test-slave-21p4"
server_key: "8068a4890c813e94b1eb7b01010a77ad"
server_key_iv: "ca707171b6a0b70e6bf98a4cfbd8af2d"
server_key_version: "encryptionkey@0"
I20260926 05:50:45.941488 31599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:45.962404 31599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:45.964633 31599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:45.964852 31599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:45.965137 31599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:45.965471 31599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:45.965524 31599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.965565 31599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:45.965593 31599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:45.971436 31599 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:36449
I20260926 05:50:45.971482 31727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:36449 every 8 connection(s)
I20260926 05:50:45.971812 31599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:50:45.973533 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31599
I20260926 05:50:45.973662 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:50:45.973879 12252 external_mini_cluster.cc:1428] Setting key aa428ea326ab14be9bc1512b2b205d87
Sep 26 05:50:45 dist-test-slave-21p4 krb5kdc[31358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.2@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.3@KRBTEST.COM; defaulting to no policy
I20260926 05:50:45.982317 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:45.972562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:59203 (local address 127.11.247.62:37109)
0926 05:50:45.972633 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:45.972635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:45.972865 (+ 230us) server_negotiation.cc:407] Connection header received
0926 05:50:45.973428 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:45.973430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:45.973443 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:45.973497 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:45.974767 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.975274 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.976708 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.976917 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.976979 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:45.979436 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:45.979438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:45.979439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:45.979448 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:45.980875 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:45.981297 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:45.981298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:45.981299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:45.981366 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:45.981622 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:45.981623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:45.981623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:45.981745 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:45.981821 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:45.982135 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 05:50:45.982223 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:50:45.982509 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:45.972528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37109 (local address 127.11.247.2:59203)
0926 05:50:45.972719 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:45.972767 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:45.973320 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:45.973576 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:45.973580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:45.974354 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:45.974673 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:45.974679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.975382 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.975383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:45.976611 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:45.976615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:45.976992 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:45.976993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:45.977144 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:45.977652 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:45.977662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:45.979333 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:45.980977 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:45.980985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:45.980988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:45.981218 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:45.981440 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:45.981441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:45.981441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:45.981554 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:45.981813 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:45.981819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:45.981993 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 05:50:45.982102 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:50:45.982270 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260926 05:50:45.983065 31728 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37109
I20260926 05:50:45.983152 31728 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:45.983302 31728 heartbeater.cc:511] Master 127.11.247.62:37109 requested a full tablet report, sending...
I20260926 05:50:45.983656 31403 ts_manager.cc:194] Registered new tserver with Master: b3343ebecd744ccfba349ef291ec0ade (127.11.247.2:36449)
I20260926 05:50:45.984119 31403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.2@KRBTEST.COM'} at 127.11.247.2:59203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
Entry for principal kudu/127.11.247.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab.
I20260926 05:50:46.001224 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37109
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:46.083061 31733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:46.083308 31733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:46.083348 31733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:46.084512 31733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:46.084579 31733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:46.084625 31733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:46.086130 31733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.3.keytab
--principal=kudu/127.11.247.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:46.086414 31733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:46.086707 31733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:46.091840 31738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:46.091848 31741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.092042 31733 server_base.cc:1034] running on GCE node
W20260926 05:50:46.091848 31739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.092330 31733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:46.092573 31733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:46.093786 31733 hybrid_clock.cc:648] HybridClock initialized: now 1790401846093762 us; error 34 us; skew 500 ppm
Sep 26 05:50:46 dist-test-slave-21p4 krb5kdc[31358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401846, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:46.096576 31733 init.cc:377] Logged in from keytab as kudu/127.11.247.3@KRBTEST.COM (short username kudu)
I20260926 05:50:46.096992 31733 webserver.cc:460] Webserver started at http://127.11.247.3:39457/ using document root <none> and password file <none>
I20260926 05:50:46.097198 31733 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:46.097240 31733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:46.097340 31733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:46.098354 31733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "c501b4463f084cef96586353b6b9d9c2"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "8d97bf003c74f5775598e62726909a3a"
server_key_iv: "274f5c1ef188f61ff4573a76a350c5d4"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.098727 31733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "c501b4463f084cef96586353b6b9d9c2"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "8d97bf003c74f5775598e62726909a3a"
server_key_iv: "274f5c1ef188f61ff4573a76a350c5d4"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.100035 31733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:46.101606 31748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.101833 31733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:46.101924 31733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "c501b4463f084cef96586353b6b9d9c2"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "8d97bf003c74f5775598e62726909a3a"
server_key_iv: "274f5c1ef188f61ff4573a76a350c5d4"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.102001 31733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:46.113914 31733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:46.116333 31733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:46.116482 31733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:46.116768 31733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:46.117087 31733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:46.117131 31733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.117153 31733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:46.117166 31733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.123176 31733 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:44029
I20260926 05:50:46.123260 31861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:44029 every 8 connection(s)
I20260926 05:50:46.123606 31733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:50:46.126147 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31733
I20260926 05:50:46.126266 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:50:46.126470 12252 external_mini_cluster.cc:1428] Setting key a7bd952a165edf5d7fb2cc0d0cbab010
Sep 26 05:50:46 dist-test-slave-21p4 krb5kdc[31358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401846, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.3@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:50:46.133903 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.124327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:54703 (local address 127.11.247.62:37109)
0926 05:50:46.124376 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.124379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.124579 (+ 200us) server_negotiation.cc:407] Connection header received
0926 05:50:46.125118 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.125119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.125133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.125185 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.126284 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.127055 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.128226 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.128424 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.128504 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.130893 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.130895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.130895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:46.130901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.132401 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:46.132831 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:46.132832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:46.132832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:46.132892 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:46.133263 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:46.133264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:46.133265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:46.133400 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.133474 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.133728 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.133840 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:50:46.134063 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.124294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37109 (local address 127.11.247.3:54703)
0926 05:50:46.124432 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.124500 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.125004 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.125265 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.125269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.125847 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:46.126134 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.126141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.127163 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.127164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.128116 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.128118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.128511 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.128512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.128610 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.129143 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:50:46.129150 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:46.130519 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:46.132473 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:46.132480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:46.132481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:46.132687 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:46.133073 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:50:46.133074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:50:46.133074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:50:46.133179 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:50:46.133476 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:46.133481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:46.133604 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.133692 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.133819 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260926 05:50:46.134642 31862 heartbeater.cc:347] Connected to a master server at 127.11.247.62:37109
I20260926 05:50:46.134757 31862 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:46.134943 31862 heartbeater.cc:511] Master 127.11.247.62:37109 requested a full tablet report, sending...
I20260926 05:50:46.135346 31403 ts_manager.cc:194] Registered new tserver with Master: c501b4463f084cef96586353b6b9d9c2 (127.11.247.3:44029)
I20260926 05:50:46.135716 31403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.247.3@KRBTEST.COM'} at 127.11.247.3:54703
I20260926 05:50:46.138092 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:46.145658 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.139419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.11.247.62:37109)
0926 05:50:46.139523 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.139525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.139540 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:50:46.139607 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.139608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.139618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.139662 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.140196 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.140677 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.141474 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.141657 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.141716 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.142985 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.142986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.142986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:46.142993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.144635 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:46.144966 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:46.144967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:46.144967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:46.145031 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:46.145203 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:46.145203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:46.145204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:46.145347 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.145430 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.145526 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.145596 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 05:50:46.147336 31403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:46.149569 31403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:46.158226 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.152106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:36449 (local address 127.0.0.1:46678)
0926 05:50:46.152443 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.152453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.152706 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.153141 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.153142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.153144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.153208 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.153210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.154525 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.154527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.157067 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.157070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.157991 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.157992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.158055 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.158066 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.158124 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.158163 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":1399,"thread_start_us":88,"threads_started":1}
I20260926 05:50:46.158277 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.152349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.11.247.2:36449)
0926 05:50:46.152974 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.152978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.152987 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:46.153024 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.153026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.153042 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.153137 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.153291 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.154408 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.157133 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.157903 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.157985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.158046 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.158141 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.158183 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":80,"threads_started":1}
I20260926 05:50:46.158785 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.152552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.11.247.1:41117)
0926 05:50:46.152748 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.152753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.152868 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:50:46.152916 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.152918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.152935 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.153018 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.153297 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.154507 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.157219 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.157916 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.157985 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.158046 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.158658 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.158709 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260926 05:50:46.159035 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.152301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41117 (local address 127.0.0.1:39308)
0926 05:50:46.152800 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.152808 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.152853 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.153076 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.153080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.153083 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.153184 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.153187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.154717 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.154719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.157173 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.157174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.158476 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.158477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.158535 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.158538 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.158892 (+ 354us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.158940 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":5,"thread_start_us":86,"threads_started":1}
I20260926 05:50:46.159044 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.152106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44029 (local address 127.0.0.1:54990)
0926 05:50:46.152213 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.152256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.152334 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.155075 (+ 2741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.155076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.155078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.155121 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.155122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.156031 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.156033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.156958 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.156960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.158826 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.158827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.158895 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.158899 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.158929 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.158976 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":53,"threads_started":1}
I20260926 05:50:46.159298 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.154710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.11.247.3:44029)
0926 05:50:46.154904 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.154908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.154918 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:50:46.154955 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.154957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.154973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.155075 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.155176 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.155906 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.157045 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.158716 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.159100 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.159172 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.159199 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.159239 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260926 05:50:46.159158 31662 tablet_service.cc:1467] Processing CreateTablet for tablet b419b6dbbe67421d82dac72495efdea4 (DEFAULT_TABLE table=test-table [id=9334a7ea5d9a414d9e99615fb50f7429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:46.159543 31662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b419b6dbbe67421d82dac72495efdea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.159669 31528 tablet_service.cc:1467] Processing CreateTablet for tablet b419b6dbbe67421d82dac72495efdea4 (DEFAULT_TABLE table=test-table [id=9334a7ea5d9a414d9e99615fb50f7429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:46.159897 31528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b419b6dbbe67421d82dac72495efdea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.162189 31879 tablet_bootstrap.cc:492] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Bootstrap starting.
I20260926 05:50:46.162742 31880 tablet_bootstrap.cc:492] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: Bootstrap starting.
I20260926 05:50:46.162719 31796 tablet_service.cc:1467] Processing CreateTablet for tablet b419b6dbbe67421d82dac72495efdea4 (DEFAULT_TABLE table=test-table [id=9334a7ea5d9a414d9e99615fb50f7429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:46.162897 31879 tablet_bootstrap.cc:654] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.163077 31796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b419b6dbbe67421d82dac72495efdea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.163381 31879 log.cc:824] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.163390 31880 tablet_bootstrap.cc:654] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.164067 31879 tablet_bootstrap.cc:492] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: No bootstrap required, opened a new log
I20260926 05:50:46.164057 31880 log.cc:824] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.164113 31879 ts_tablet_manager.cc:1397] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:50:46.165787 31879 raft_consensus.cc:348] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.165975 31879 raft_consensus.cc:374] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.166014 31879 raft_consensus.cc:729] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3343ebecd744ccfba349ef291ec0ade, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.166023 31883 tablet_bootstrap.cc:492] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Bootstrap starting.
I20260926 05:50:46.166172 31879 consensus_queue.cc:260] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.166482 31879 ts_tablet_manager.cc:1428] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:50:46.166705 31883 tablet_bootstrap.cc:654] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.166705 31728 heartbeater.cc:503] Master 127.11.247.62:37109 was elected leader, sending a full tablet report...
I20260926 05:50:46.167127 31880 tablet_bootstrap.cc:492] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: No bootstrap required, opened a new log
I20260926 05:50:46.167217 31880 ts_tablet_manager.cc:1397] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 05:50:46.167290 31883 log.cc:824] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.168037 31883 tablet_bootstrap.cc:492] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: No bootstrap required, opened a new log
I20260926 05:50:46.168095 31883 ts_tablet_manager.cc:1397] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:50:46.168857 31880 raft_consensus.cc:348] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.169013 31880 raft_consensus.cc:374] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.169059 31880 raft_consensus.cc:729] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c66faaef564a2ca3bbc2f3136210fe, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.169174 31880 consensus_queue.cc:260] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.169483 31880 ts_tablet_manager.cc:1428] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:50:46.169598 31594 heartbeater.cc:503] Master 127.11.247.62:37109 was elected leader, sending a full tablet report...
I20260926 05:50:46.169750 31883 raft_consensus.cc:348] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.169903 31883 raft_consensus.cc:374] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.169946 31883 raft_consensus.cc:729] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c501b4463f084cef96586353b6b9d9c2, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.170063 31883 consensus_queue.cc:260] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.170329 31883 ts_tablet_manager.cc:1428] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:50:46.170384 31862 heartbeater.cc:503] Master 127.11.247.62:37109 was elected leader, sending a full tablet report...
W20260926 05:50:46.222409 31729 tablet.cc:2369] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:50:46.224525 31884 raft_consensus.cc:482] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:46.224647 31884 raft_consensus.cc:504] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.224959 31884 leader_election.cc:288] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c501b4463f084cef96586353b6b9d9c2 (127.11.247.3:44029), b7c66faaef564a2ca3bbc2f3136210fe (127.11.247.1:41117)
I20260926 05:50:46.228380 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.225397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:55657 (local address 127.11.247.1:41117)
0926 05:50:46.225446 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.225449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.225586 (+ 137us) server_negotiation.cc:407] Connection header received
0926 05:50:46.225619 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.225620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.225627 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.225674 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.225868 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.226559 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.227640 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.228018 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.228073 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.228079 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.228258 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.228310 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:50:46.228440 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.225344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:40717 (local address 127.11.247.3:44029)
0926 05:50:46.225399 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.225402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.225412 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:50:46.225443 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.225444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.225455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.225518 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.225729 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.226517 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.227617 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.228038 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.228136 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.228145 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.228329 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.228377 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:50:46.228451 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.225191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:44029 (local address 127.11.247.2:40717)
0926 05:50:46.225259 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.225273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.225377 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.225536 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.225538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.225539 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.225634 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.225636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.226616 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.226617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.227522 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.227525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.228156 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.228157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.228243 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.228247 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.228306 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.228342 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":4}
I20260926 05:50:46.228638 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.225310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41117 (local address 127.11.247.2:55657)
0926 05:50:46.225487 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.225496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.225546 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.225702 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.225703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.225705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.225777 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.225778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.226666 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.226667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.227545 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.227548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.228102 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.228103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.228178 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.228182 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.228237 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.228266 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260926 05:50:46.228832 31816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b419b6dbbe67421d82dac72495efdea4" candidate_uuid: "b3343ebecd744ccfba349ef291ec0ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c501b4463f084cef96586353b6b9d9c2" is_pre_election: true
I20260926 05:50:46.228955 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b419b6dbbe67421d82dac72495efdea4" candidate_uuid: "b3343ebecd744ccfba349ef291ec0ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" is_pre_election: true
I20260926 05:50:46.229033 31816 raft_consensus.cc:2454] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3343ebecd744ccfba349ef291ec0ade in term 0.
I20260926 05:50:46.229131 31548 raft_consensus.cc:2454] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3343ebecd744ccfba349ef291ec0ade in term 0.
I20260926 05:50:46.229292 31618 leader_election.cc:302] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3343ebecd744ccfba349ef291ec0ade, c501b4463f084cef96586353b6b9d9c2; no voters:
I20260926 05:50:46.229436 31884 raft_consensus.cc:2781] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:46.229493 31884 raft_consensus.cc:482] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:46.229518 31884 raft_consensus.cc:3037] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.230221 31884 raft_consensus.cc:504] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.230365 31884 leader_election.cc:288] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [CANDIDATE]: Term 1 election: Requested vote from peers c501b4463f084cef96586353b6b9d9c2 (127.11.247.3:44029), b7c66faaef564a2ca3bbc2f3136210fe (127.11.247.1:41117)
I20260926 05:50:46.230508 31816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b419b6dbbe67421d82dac72495efdea4" candidate_uuid: "b3343ebecd744ccfba349ef291ec0ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c501b4463f084cef96586353b6b9d9c2"
I20260926 05:50:46.230540 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b419b6dbbe67421d82dac72495efdea4" candidate_uuid: "b3343ebecd744ccfba349ef291ec0ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c66faaef564a2ca3bbc2f3136210fe"
I20260926 05:50:46.230628 31816 raft_consensus.cc:3037] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.230670 31548 raft_consensus.cc:3037] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.231463 31816 raft_consensus.cc:2454] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3343ebecd744ccfba349ef291ec0ade in term 1.
I20260926 05:50:46.231586 31548 raft_consensus.cc:2454] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3343ebecd744ccfba349ef291ec0ade in term 1.
I20260926 05:50:46.231647 31618 leader_election.cc:302] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3343ebecd744ccfba349ef291ec0ade, c501b4463f084cef96586353b6b9d9c2; no voters:
I20260926 05:50:46.231745 31884 raft_consensus.cc:2781] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:46.231905 31884 raft_consensus.cc:686] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 1 LEADER]: Becoming Leader. State: Replica: b3343ebecd744ccfba349ef291ec0ade, State: Running, Role: LEADER
I20260926 05:50:46.232007 31884 consensus_queue.cc:237] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } }
I20260926 05:50:46.233031 31403 catalog_manager.cc:5495] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade reported cstate change: term changed from 0 to 1, leader changed from <none> to b3343ebecd744ccfba349ef291ec0ade (127.11.247.2). New cstate: current_term: 1 leader_uuid: "b3343ebecd744ccfba349ef291ec0ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3343ebecd744ccfba349ef291ec0ade" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 36449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260926 05:50:46.250499 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.248005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.11.247.2:36449)
0926 05:50:46.248070 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.248072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.248081 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:46.248142 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.248143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.248163 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.248225 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:46.248361 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.248875 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.249644 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.249847 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.249929 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.250117 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:46.250223 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:46.250326 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.250344 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.250368 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:50:46.252511 31548 raft_consensus.cc:1261] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 1 FOLLOWER]: Refusing update from remote peer b3343ebecd744ccfba349ef291ec0ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:46.252508 31816 raft_consensus.cc:1261] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 1 FOLLOWER]: Refusing update from remote peer b3343ebecd744ccfba349ef291ec0ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:46.252884 31884 consensus_queue.cc:1035] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "c501b4463f084cef96586353b6b9d9c2" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:46.253091 31889 consensus_queue.cc:1035] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7c66faaef564a2ca3bbc2f3136210fe" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:46.256296 31893 mvcc.cc:205] Tried to move back new op lower bound from 7333485962246864896 to 7333485962166767616. Current Snapshot: MvccSnapshot[applied={T|T < 7333485962246864896}]
I20260926 05:50:46.258939 31402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:46.259063 31402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:46.260027 31402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b61259e5b1c34996b644452e0800a899: Sending DeleteTablet for 3 replicas of tablet b419b6dbbe67421d82dac72495efdea4
I20260926 05:50:46.260426 31528 tablet_service.cc:1514] Processing DeleteTablet for tablet b419b6dbbe67421d82dac72495efdea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:46 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:39308
I20260926 05:50:46.260447 31796 tablet_service.cc:1514] Processing DeleteTablet for tablet b419b6dbbe67421d82dac72495efdea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:46 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:54990
I20260926 05:50:46.260726 31902 tablet_replica.cc:330] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: stopping tablet replica
I20260926 05:50:46.260725 31662 tablet_service.cc:1514] Processing DeleteTablet for tablet b419b6dbbe67421d82dac72495efdea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:46 UTC) from {username='kudu', principal='kudu/127.11.247.62@KRBTEST.COM'} at 127.0.0.1:46678
I20260926 05:50:46.260749 31903 tablet_replica.cc:330] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: stopping tablet replica
I20260926 05:50:46.260849 31902 raft_consensus.cc:2229] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:46.260970 31902 raft_consensus.cc:2258] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:46.260982 31903 raft_consensus.cc:2229] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:46.261170 31903 raft_consensus.cc:2258] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:46.261224 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31465
I20260926 05:50:46.261461 31902 ts_tablet_manager.cc:1905] T b419b6dbbe67421d82dac72495efdea4 P b7c66faaef564a2ca3bbc2f3136210fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:46.261828 31904 tablet_replica.cc:330] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: stopping tablet replica
I20260926 05:50:46.261924 31904 raft_consensus.cc:2229] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:50:46.262046 31904 raft_consensus.cc:2258] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:46.262501 31904 ts_tablet_manager.cc:1905] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:46.262758 31903 ts_tablet_manager.cc:1905] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:46.264770 31903 ts_tablet_manager.cc:1918] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:46.264770 31904 ts_tablet_manager.cc:1918] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:46.264853 31904 log.cc:1196] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/b419b6dbbe67421d82dac72495efdea4
I20260926 05:50:46.264985 31903 log.cc:1196] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/b419b6dbbe67421d82dac72495efdea4
I20260926 05:50:46.265184 31904 ts_tablet_manager.cc:1939] T b419b6dbbe67421d82dac72495efdea4 P b3343ebecd744ccfba349ef291ec0ade: Deleting consensus metadata
I20260926 05:50:46.265384 31903 ts_tablet_manager.cc:1939] T b419b6dbbe67421d82dac72495efdea4 P c501b4463f084cef96586353b6b9d9c2: Deleting consensus metadata
I20260926 05:50:46.265676 31389 catalog_manager.cc:4841] TS b3343ebecd744ccfba349ef291ec0ade (127.11.247.2:36449): tablet b419b6dbbe67421d82dac72495efdea4 (table test-table [id=9334a7ea5d9a414d9e99615fb50f7429]) successfully deleted
I20260926 05:50:46.265823 31390 catalog_manager.cc:4841] TS c501b4463f084cef96586353b6b9d9c2 (127.11.247.3:44029): tablet b419b6dbbe67421d82dac72495efdea4 (table test-table [id=9334a7ea5d9a414d9e99615fb50f7429]) successfully deleted
W20260926 05:50:46.269783 31390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:41117 (error 108)
I20260926 05:50:46.270030 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31599
I20260926 05:50:46.271159 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.271015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:41117 (local address 127.0.0.1:39324)
0926 05:50:46.271074 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.271113 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:41117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 05:50:46.271284 31390 catalog_manager.cc:4584] TS b7c66faaef564a2ca3bbc2f3136210fe (127.11.247.1:41117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b419b6dbbe67421d82dac72495efdea4: Network error: Client connection negotiation failed: client connection to 127.11.247.1:41117: connect: Connection refused (error 111)
I20260926 05:50:46.274777 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31733
I20260926 05:50:46.279451 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31371
2026-09-26T05:50:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1999 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T05:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:46Z Disabled control of system clock
I20260926 05:50:46.307775 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:35383
--webserver_interface=127.11.247.62
--webserver_port=0
--builtin_ntp_servers=127.11.247.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:46.389184 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:46.389423 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:46.389446 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:46.390543 31909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:46.390620 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:46.390636 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:46.390650 31909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:46.390661 31909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:46.392127 31909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:35383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:46.392334 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:46.392571 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:46.397612 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:46.397600 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.397800 31909 server_base.cc:1034] running on GCE node
W20260926 05:50:46.397600 31917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.398208 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:46.398571 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:46.399887 31909 hybrid_clock.cc:648] HybridClock initialized: now 1790401846399862 us; error 71 us; skew 500 ppm
I20260926 05:50:46.401515 31909 webserver.cc:460] Webserver started at http://127.11.247.62:35213/ using document root <none> and password file <none>
I20260926 05:50:46.401813 31909 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:46.401888 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:46.402047 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:46.403265 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "2f5f7ed8f58144bbbd743584a995a124"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "d6d752bb3720fc6cd05e6f9c02fa817a"
server_key_iv: "54a85a80addc750d9784931d7607ad03"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.403635 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "2f5f7ed8f58144bbbd743584a995a124"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "d6d752bb3720fc6cd05e6f9c02fa817a"
server_key_iv: "54a85a80addc750d9784931d7607ad03"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.404971 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:46.405855 31923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.406059 31909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:46.406170 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "2f5f7ed8f58144bbbd743584a995a124"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "d6d752bb3720fc6cd05e6f9c02fa817a"
server_key_iv: "54a85a80addc750d9784931d7607ad03"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.406245 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:46.423705 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:46.424052 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:46.424217 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:46.428367 31909 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:35383
I20260926 05:50:46.428406 31975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:35383 every 8 connection(s)
I20260926 05:50:46.428956 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:46.429991 31976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.432515 31976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124: Bootstrap starting.
I20260926 05:50:46.432560 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 31909
I20260926 05:50:46.432667 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:46.432853 12252 external_mini_cluster.cc:1428] Setting key fcfd78911d0ad646fa7445b628d0ab50
I20260926 05:50:46.433287 31976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.433666 31976 log.cc:824] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.434451 31976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124: No bootstrap required, opened a new log
I20260926 05:50:46.436798 31976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f7ed8f58144bbbd743584a995a124" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35383 } }
I20260926 05:50:46.437000 31976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.437036 31976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5f7ed8f58144bbbd743584a995a124, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.437160 31976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f7ed8f58144bbbd743584a995a124" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35383 } }
I20260926 05:50:46.437228 31976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:46.437258 31976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:46.437300 31976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.438007 31976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f7ed8f58144bbbd743584a995a124" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35383 } }
I20260926 05:50:46.438130 31976 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5f7ed8f58144bbbd743584a995a124; no voters:
I20260926 05:50:46.438306 31976 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:46.438373 31981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:46.438515 31981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [term 1 LEADER]: Becoming Leader. State: Replica: 2f5f7ed8f58144bbbd743584a995a124, State: Running, Role: LEADER
I20260926 05:50:46.438624 31981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f7ed8f58144bbbd743584a995a124" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35383 } }
I20260926 05:50:46.438628 31976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:46.439065 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5f7ed8f58144bbbd743584a995a124. Latest consensus state: current_term: 1 leader_uuid: "2f5f7ed8f58144bbbd743584a995a124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f7ed8f58144bbbd743584a995a124" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35383 } } }
I20260926 05:50:46.439154 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:46.438995 31982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5f7ed8f58144bbbd743584a995a124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5f7ed8f58144bbbd743584a995a124" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35383 } } }
I20260926 05:50:46.439246 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.433429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.11.247.62:35383)
0926 05:50:46.433656 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.433664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.433688 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:50:46.434168 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.434174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.434234 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.434354 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.434658 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.436092 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.436872 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.437479 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.437607 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.438303 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.438307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.438308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:46.438525 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.438565 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:46.438602 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.438698 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.438802 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.438902 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260926 05:50:46.439316 31982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:46.439417 31986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:46.440022 31986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:46.441715 31986 catalog_manager.cc:1347] Generated new cluster ID: 9d37189ca16248c09c5920349265a099
I20260926 05:50:46.441782 31986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:46.481029 31986 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:46.481638 31986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:46.487075 31986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124: Generated new TSK 0
I20260926 05:50:46.487368 31986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:46.492813 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.1:0
--local_ip_for_outbound_sockets=127.11.247.1
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:35383
--builtin_ntp_servers=127.11.247.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:50:46.595259 32000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:46.595464 32000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:46.595489 32000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:46.596668 32000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:46.596709 32000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:46.596733 32000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.1
I20260926 05:50:46.598151 32000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.247.1
--webserver_port=0
--tserver_master_addrs=127.11.247.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.1
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:46.598438 32000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:46.598794 32000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:46.604252 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:46.604249 32005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:46.604274 32006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.604454 32000 server_base.cc:1034] running on GCE node
I20260926 05:50:46.604669 32000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:46.604915 32000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:46.606056 32000 hybrid_clock.cc:648] HybridClock initialized: now 1790401846606044 us; error 28 us; skew 500 ppm
I20260926 05:50:46.607426 32000 webserver.cc:460] Webserver started at http://127.11.247.1:39199/ using document root <none> and password file <none>
I20260926 05:50:46.607677 32000 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:46.607728 32000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:46.607892 32000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:46.609156 32000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data/instance:
uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "67c799ad41f87252025596af7c29d193"
server_key_iv: "e6b4e4b78a4b83ab6819afca48695f83"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.609516 32000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance:
uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "67c799ad41f87252025596af7c29d193"
server_key_iv: "e6b4e4b78a4b83ab6819afca48695f83"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.611050 32000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:50:46.611804 32014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.611964 32000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:50:46.612066 32000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal
uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "67c799ad41f87252025596af7c29d193"
server_key_iv: "e6b4e4b78a4b83ab6819afca48695f83"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.612152 32000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:46.627926 32000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:46.628283 32000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:46.628494 32000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:46.628770 32000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:46.629125 32000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:46.629181 32000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.629225 32000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:46.629249 32000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.635486 32000 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.1:36093
I20260926 05:50:46.635533 32127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.1:36093 every 8 connection(s)
I20260926 05:50:46.635874 32000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/data/info.pb
I20260926 05:50:46.638542 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 32000
I20260926 05:50:46.638684 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-0/wal/instance
I20260926 05:50:46.638927 12252 external_mini_cluster.cc:1428] Setting key 4dedb3876bd25878287fbc855603fbb9
I20260926 05:50:46.639982 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.2:0
--local_ip_for_outbound_sockets=127.11.247.2
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:35383
--builtin_ntp_servers=127.11.247.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:50:46.642119 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.636579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.1:51047 (local address 127.11.247.62:35383)
0926 05:50:46.636623 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.636626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.636894 (+ 268us) server_negotiation.cc:407] Connection header received
0926 05:50:46.637444 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.637445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.637458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.637517 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.638250 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.638986 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.640514 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.640860 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.640936 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.641855 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.641856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.641857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:46.641861 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.641875 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:46.641879 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.641925 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.642036 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.642061 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:50:46.642423 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.636566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:35383 (local address 127.11.247.1:51047)
0926 05:50:46.636726 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.636760 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.637335 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.637592 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.637596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.637604 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:46.638139 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.638144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.639132 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.639133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.640444 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.640447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.640946 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.640947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.641055 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.641681 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:46.641682 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:46.641724 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:46.641754 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:46.641946 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:46.641951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:46.641954 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.642029 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.642146 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260926 05:50:46.643021 32128 heartbeater.cc:347] Connected to a master server at 127.11.247.62:35383
I20260926 05:50:46.643121 32128 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:46.643318 32128 heartbeater.cc:511] Master 127.11.247.62:35383 requested a full tablet report, sending...
I20260926 05:50:46.643855 31940 ts_manager.cc:194] Registered new tserver with Master: ae69a4be7c1e4e9f83d1cea22a3c6632 (127.11.247.1:36093)
I20260926 05:50:46.644520 31940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.1:51047
W20260926 05:50:46.722510 32131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:46.722790 32131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:46.722836 32131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:46.724022 32131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:46.724098 32131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:46.724143 32131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.2
I20260926 05:50:46.725627 32131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.247.2
--webserver_port=0
--tserver_master_addrs=127.11.247.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.2
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:46.725919 32131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:46.726236 32131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:46.731338 32137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.731436 32131 server_base.cc:1034] running on GCE node
W20260926 05:50:46.731315 32139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:46.731315 32136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.731818 32131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:46.732105 32131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:46.733261 32131 hybrid_clock.cc:648] HybridClock initialized: now 1790401846733240 us; error 37 us; skew 500 ppm
I20260926 05:50:46.734760 32131 webserver.cc:460] Webserver started at http://127.11.247.2:39389/ using document root <none> and password file <none>
I20260926 05:50:46.735013 32131 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:46.735060 32131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:46.735205 32131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:46.736328 32131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data/instance:
uuid: "fa22f0a779de439faaebb5201b57ee6a"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "73e2523183bf65f191fd0a5bd089152e"
server_key_iv: "870aa779152f80485614be4f8b9a6bb6"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.736720 32131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance:
uuid: "fa22f0a779de439faaebb5201b57ee6a"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "73e2523183bf65f191fd0a5bd089152e"
server_key_iv: "870aa779152f80485614be4f8b9a6bb6"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.738251 32131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:46.739159 32145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.739370 32131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:50:46.739487 32131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal
uuid: "fa22f0a779de439faaebb5201b57ee6a"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "73e2523183bf65f191fd0a5bd089152e"
server_key_iv: "870aa779152f80485614be4f8b9a6bb6"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.739595 32131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:46.755353 32131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:46.755698 32131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:46.755852 32131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:46.756093 32131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:46.756433 32131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:46.756464 32131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.756516 32131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:46.756549 32131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.762364 32131 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.2:35317
I20260926 05:50:46.762447 32258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.2:35317 every 8 connection(s)
I20260926 05:50:46.762807 32131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/data/info.pb
I20260926 05:50:46.765843 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 32131
I20260926 05:50:46.765974 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal/instance
I20260926 05:50:46.766224 12252 external_mini_cluster.cc:1428] Setting key 59c8781ba9954fdbbbd72071faa33f04
I20260926 05:50:46.767235 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.247.3:0
--local_ip_for_outbound_sockets=127.11.247.3
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:35383
--builtin_ntp_servers=127.11.247.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:50:46.768929 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.763586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.2:47251 (local address 127.11.247.62:35383)
0926 05:50:46.763648 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.763650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.763916 (+ 266us) server_negotiation.cc:407] Connection header received
0926 05:50:46.764461 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.764462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.764474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.764519 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.765245 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.765868 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.767327 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.767657 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.767734 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.768658 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.768659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.768660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:46.768664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.768677 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:46.768681 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.768736 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.768840 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.768864 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:50:46.769212 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.763573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:35383 (local address 127.11.247.2:47251)
0926 05:50:46.763771 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.763805 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.764342 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.764601 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.764605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.764613 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:46.765149 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.765154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.765984 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.765985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.767220 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.767223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.767760 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.767762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.767886 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.768481 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:46.768482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:46.768527 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:46.768562 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:46.768753 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:46.768759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:46.768762 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.768834 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.768951 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260926 05:50:46.769774 32259 heartbeater.cc:347] Connected to a master server at 127.11.247.62:35383
I20260926 05:50:46.769889 32259 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:46.770072 32259 heartbeater.cc:511] Master 127.11.247.62:35383 requested a full tablet report, sending...
I20260926 05:50:46.770459 31940 ts_manager.cc:194] Registered new tserver with Master: fa22f0a779de439faaebb5201b57ee6a (127.11.247.2:35317)
I20260926 05:50:46.771013 31940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.2:47251
W20260926 05:50:46.849459 32262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:46.849676 32262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:46.849699 32262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:46.850960 32262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:46.851020 32262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:46.851050 32262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.247.3
I20260926 05:50:46.852389 32262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.247.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.247.3
--webserver_port=0
--tserver_master_addrs=127.11.247.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.247.3
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:46.852588 32262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:46.852845 32262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:46.858028 32268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:46.858036 32267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.858088 32262 server_base.cc:1034] running on GCE node
W20260926 05:50:46.858036 32270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:46.858517 32262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:46.858798 32262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:46.859956 32262 hybrid_clock.cc:648] HybridClock initialized: now 1790401846859944 us; error 29 us; skew 500 ppm
I20260926 05:50:46.861343 32262 webserver.cc:460] Webserver started at http://127.11.247.3:44799/ using document root <none> and password file <none>
I20260926 05:50:46.861610 32262 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:46.861680 32262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:46.861868 32262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:46.863068 32262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data/instance:
uuid: "75e9acd03f31473cbf77cb463e38eb30"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "4292b74e2fccad452fc10439c4a0abbd"
server_key_iv: "9a4db775ab53e07d90bb627dff4ed40e"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.863435 32262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance:
uuid: "75e9acd03f31473cbf77cb463e38eb30"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "4292b74e2fccad452fc10439c4a0abbd"
server_key_iv: "9a4db775ab53e07d90bb627dff4ed40e"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.864883 32262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:46.865800 32276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.866029 32262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:46.866124 32262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal
uuid: "75e9acd03f31473cbf77cb463e38eb30"
format_stamp: "Formatted at 2026-09-26 05:50:46 on dist-test-slave-21p4"
server_key: "4292b74e2fccad452fc10439c4a0abbd"
server_key_iv: "9a4db775ab53e07d90bb627dff4ed40e"
server_key_version: "encryptionkey@0"
I20260926 05:50:46.866187 32262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:46.890823 32262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:46.891140 32262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:46.891260 32262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:46.891458 32262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:50:46.891871 32262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:50:46.891914 32262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.891938 32262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:50:46.891990 32262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:46.897737 32262 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.3:32831
I20260926 05:50:46.897785 32389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.3:32831 every 8 connection(s)
I20260926 05:50:46.898132 32262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/data/info.pb
I20260926 05:50:46.903101 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 32262
I20260926 05:50:46.903227 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal/instance
I20260926 05:50:46.903493 12252 external_mini_cluster.cc:1428] Setting key 68b89d6405e6876f05eb2e13ee8a8197
I20260926 05:50:46.904274 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.899079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:43461 (local address 127.11.247.62:35383)
0926 05:50:46.899143 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.899145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.899351 (+ 206us) server_negotiation.cc:407] Connection header received
0926 05:50:46.899985 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.899987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.899998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.900068 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.900913 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.901430 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.902488 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.902701 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.902808 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.903916 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.903918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.903918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:46.903923 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.903967 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:46.903972 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.904025 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.904145 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.904183 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:50:46.904549 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.898998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:35383 (local address 127.11.247.3:43461)
0926 05:50:46.899198 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.899241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.899847 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.900180 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.900184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.900193 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:50:46.900727 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.900731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.901549 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.901550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.902399 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.902401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.902864 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.902866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.903031 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.903700 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:50:46.903701 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:50:46.903743 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:50:46.903794 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:50:46.904043 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:50:46.904050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:50:46.904053 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.904143 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.904276 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260926 05:50:46.905055 32390 heartbeater.cc:347] Connected to a master server at 127.11.247.62:35383
I20260926 05:50:46.905151 32390 heartbeater.cc:464] Registering TS with master...
I20260926 05:50:46.905297 32390 heartbeater.cc:511] Master 127.11.247.62:35383 requested a full tablet report, sending...
I20260926 05:50:46.905664 31940 ts_manager.cc:194] Registered new tserver with Master: 75e9acd03f31473cbf77cb463e38eb30 (127.11.247.3:32831)
I20260926 05:50:46.906047 31940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.247.3:43461
I20260926 05:50:46.915583 12252 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:50:46.920231 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.917066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.11.247.62:35383)
0926 05:50:46.917113 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.917116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.917231 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:50:46.917312 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.917313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.917323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.917408 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:46.917588 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.918090 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.918944 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.919156 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.919235 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.919954 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:46.919955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:46.919955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:46.919961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:46.919979 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:46.919987 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:46.920048 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.920108 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.920134 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:50:46.921962 31940 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:50:46.923609 31940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:50:46.930740 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.925800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.3:32831 (local address 127.0.0.1:44080)
0926 05:50:46.926281 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.926326 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.926423 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.926893 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.926893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.926894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.926920 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.926921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.928145 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.928147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.929330 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.929334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.930366 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.930367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.930466 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.930489 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.930542 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.930626 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":23,"thread_start_us":107,"threads_started":1}
I20260926 05:50:46.930764 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.925828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35317 (local address 127.0.0.1:54110)
0926 05:50:46.926398 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.926408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.926459 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.926721 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.926724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.926726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.926804 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.926806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.928133 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.928134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.929309 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.929311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.930371 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.930371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.930462 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.930476 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.930545 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.930637 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":60,"thread_start_us":92,"threads_started":1}
I20260926 05:50:46.931038 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.926046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36093 (local address 127.0.0.1:51480)
0926 05:50:46.926538 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.926545 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.926643 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.926880 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.926881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.926884 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.926918 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.926919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.927890 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.927891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.929307 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.929311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.930369 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.930370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.930462 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.930476 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.930545 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.930636 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":10,"thread_start_us":68,"threads_started":1}
I20260926 05:50:46.930819 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.926151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.11.247.1:36093)
0926 05:50:46.926317 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.926322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.926658 (+ 336us) server_negotiation.cc:407] Connection header received
0926 05:50:46.926709 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.926711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.926727 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.926808 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.926993 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.927794 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.929413 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.930160 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.930231 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.930320 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.930690 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.930734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260926 05:50:46.931190 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.926023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.11.247.3:32831)
0926 05:50:46.926204 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.926209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.926388 (+ 179us) server_negotiation.cc:407] Connection header received
0926 05:50:46.926528 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.926530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.926545 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.926806 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.926976 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.928040 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.929439 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.930158 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.930226 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.930319 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.931069 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.931114 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260926 05:50:46.931344 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.926227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.11.247.2:35317)
0926 05:50:46.926382 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.926386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.926449 (+ 63us) server_negotiation.cc:407] Connection header received
0926 05:50:46.926526 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.926528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.926545 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.926658 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.926875 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.928057 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.929413 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.930160 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.930231 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.930319 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.930942 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.931211 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260926 05:50:46.931910 32324 tablet_service.cc:1467] Processing CreateTablet for tablet 8722ebb4c3f446659fcebcebd6adb2fc (DEFAULT_TABLE table=test-table [id=48cedaf74a1b41099369acc4b3ca2ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:46.932085 32193 tablet_service.cc:1467] Processing CreateTablet for tablet 8722ebb4c3f446659fcebcebd6adb2fc (DEFAULT_TABLE table=test-table [id=48cedaf74a1b41099369acc4b3ca2ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:46.932317 32324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8722ebb4c3f446659fcebcebd6adb2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.932420 32193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8722ebb4c3f446659fcebcebd6adb2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.935833 32404 tablet_bootstrap.cc:492] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Bootstrap starting.
I20260926 05:50:46.935849 32405 tablet_bootstrap.cc:492] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Bootstrap starting.
I20260926 05:50:46.936502 32404 tablet_bootstrap.cc:654] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.936504 32405 tablet_bootstrap.cc:654] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.936791 32405 log.cc:824] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.936791 32404 log.cc:824] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.937655 32404 tablet_bootstrap.cc:492] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: No bootstrap required, opened a new log
I20260926 05:50:46.937728 32404 ts_tablet_manager.cc:1397] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:50:46.937896 32405 tablet_bootstrap.cc:492] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: No bootstrap required, opened a new log
I20260926 05:50:46.938006 32405 ts_tablet_manager.cc:1397] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:50:46.939634 32062 tablet_service.cc:1467] Processing CreateTablet for tablet 8722ebb4c3f446659fcebcebd6adb2fc (DEFAULT_TABLE table=test-table [id=48cedaf74a1b41099369acc4b3ca2ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:50:46.940016 32062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8722ebb4c3f446659fcebcebd6adb2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:46.940665 32404 raft_consensus.cc:348] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.940840 32404 raft_consensus.cc:374] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.941076 32405 raft_consensus.cc:348] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.940882 32404 raft_consensus.cc:729] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e9acd03f31473cbf77cb463e38eb30, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.941303 32405 raft_consensus.cc:374] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.941425 32405 raft_consensus.cc:729] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa22f0a779de439faaebb5201b57ee6a, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.941442 32404 consensus_queue.cc:260] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.941556 32405 consensus_queue.cc:260] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.941709 32404 ts_tablet_manager.cc:1428] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 05:50:46.941825 32405 ts_tablet_manager.cc:1428] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 05:50:46.941846 32390 heartbeater.cc:503] Master 127.11.247.62:35383 was elected leader, sending a full tablet report...
I20260926 05:50:46.941942 32259 heartbeater.cc:503] Master 127.11.247.62:35383 was elected leader, sending a full tablet report...
I20260926 05:50:46.943630 32410 tablet_bootstrap.cc:492] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632: Bootstrap starting.
I20260926 05:50:46.944459 32410 tablet_bootstrap.cc:654] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:46.944854 32410 log.cc:824] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:46.946652 32410 tablet_bootstrap.cc:492] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632: No bootstrap required, opened a new log
I20260926 05:50:46.946722 32410 ts_tablet_manager.cc:1397] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:50:46.948558 32410 raft_consensus.cc:348] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.948701 32410 raft_consensus.cc:374] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:46.948724 32410 raft_consensus.cc:729] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae69a4be7c1e4e9f83d1cea22a3c6632, State: Initialized, Role: FOLLOWER
I20260926 05:50:46.948871 32410 consensus_queue.cc:260] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.949138 32410 ts_tablet_manager.cc:1428] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:50:46.949195 32128 heartbeater.cc:503] Master 127.11.247.62:35383 was elected leader, sending a full tablet report...
I20260926 05:50:46.958858 32408 raft_consensus.cc:482] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:50:46.959098 32408 raft_consensus.cc:504] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.959533 32408 leader_election.cc:288] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae69a4be7c1e4e9f83d1cea22a3c6632 (127.11.247.1:36093), fa22f0a779de439faaebb5201b57ee6a (127.11.247.2:35317)
I20260926 05:50:46.963479 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.959749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:52137 (local address 127.11.247.1:36093)
0926 05:50:46.959805 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.959807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.959820 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:50:46.959849 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.959850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.959862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.959927 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.960136 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.960901 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.962490 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.963119 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.963193 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.963202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.963364 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.963410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:50:46.963485 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.959686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36093 (local address 127.11.247.3:52137)
0926 05:50:46.959731 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.959744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.959794 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.959955 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.959957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.959958 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.960032 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.960036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.960991 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.960991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.962384 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.962388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.963190 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.963191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.963282 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.963286 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.963346 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.963393 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":98}
I20260926 05:50:46.963613 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.959829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.247.3:51305 (local address 127.11.247.2:35317)
0926 05:50:46.959909 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.959911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.959942 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:50:46.960016 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.960017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.960025 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.960066 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:50:46.960229 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.960906 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.962491 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.963119 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.963193 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.963202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.963475 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.963522 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:50:46.963822 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.959687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.2:35317 (local address 127.11.247.3:51305)
0926 05:50:46.959894 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.959902 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:50:46.959951 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:50:46.960088 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:50:46.960089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:50:46.960090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:50:46.960165 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.960166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.960991 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.960991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.962385 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:50:46.962388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.963252 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.963256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:50:46.963377 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.963382 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:50:46.963705 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 05:50:46.963745 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":98,"thread_start_us":106,"threads_started":1}
I20260926 05:50:46.964149 32082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8722ebb4c3f446659fcebcebd6adb2fc" candidate_uuid: "75e9acd03f31473cbf77cb463e38eb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" is_pre_election: true
I20260926 05:50:46.964179 32213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8722ebb4c3f446659fcebcebd6adb2fc" candidate_uuid: "75e9acd03f31473cbf77cb463e38eb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa22f0a779de439faaebb5201b57ee6a" is_pre_election: true
I20260926 05:50:46.964303 32082 raft_consensus.cc:2454] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e9acd03f31473cbf77cb463e38eb30 in term 0.
I20260926 05:50:46.964330 32213 raft_consensus.cc:2454] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e9acd03f31473cbf77cb463e38eb30 in term 0.
I20260926 05:50:46.964563 32277 leader_election.cc:302] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e9acd03f31473cbf77cb463e38eb30, ae69a4be7c1e4e9f83d1cea22a3c6632; no voters:
I20260926 05:50:46.964710 32408 raft_consensus.cc:2781] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:50:46.964772 32408 raft_consensus.cc:482] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:50:46.964790 32408 raft_consensus.cc:3037] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.965471 32408 raft_consensus.cc:504] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.965616 32408 leader_election.cc:288] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [CANDIDATE]: Term 1 election: Requested vote from peers ae69a4be7c1e4e9f83d1cea22a3c6632 (127.11.247.1:36093), fa22f0a779de439faaebb5201b57ee6a (127.11.247.2:35317)
I20260926 05:50:46.965790 32213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8722ebb4c3f446659fcebcebd6adb2fc" candidate_uuid: "75e9acd03f31473cbf77cb463e38eb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa22f0a779de439faaebb5201b57ee6a"
I20260926 05:50:46.965801 32082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8722ebb4c3f446659fcebcebd6adb2fc" candidate_uuid: "75e9acd03f31473cbf77cb463e38eb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632"
I20260926 05:50:46.965871 32213 raft_consensus.cc:3037] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.965904 32082 raft_consensus.cc:3037] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:46.966830 32213 raft_consensus.cc:2454] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e9acd03f31473cbf77cb463e38eb30 in term 1.
I20260926 05:50:46.966830 32082 raft_consensus.cc:2454] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e9acd03f31473cbf77cb463e38eb30 in term 1.
I20260926 05:50:46.967026 32278 leader_election.cc:302] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e9acd03f31473cbf77cb463e38eb30, fa22f0a779de439faaebb5201b57ee6a; no voters:
I20260926 05:50:46.967150 32408 raft_consensus.cc:2781] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:46.967312 32408 raft_consensus.cc:686] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 1 LEADER]: Becoming Leader. State: Replica: 75e9acd03f31473cbf77cb463e38eb30, State: Running, Role: LEADER
I20260926 05:50:46.967415 32408 consensus_queue.cc:237] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } }
I20260926 05:50:46.968294 31939 catalog_manager.cc:5495] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e9acd03f31473cbf77cb463e38eb30 (127.11.247.3). New cstate: current_term: 1 leader_uuid: "75e9acd03f31473cbf77cb463e38eb30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e9acd03f31473cbf77cb463e38eb30" member_type: VOTER last_known_addr { host: "127.11.247.3" port: 32831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 } health_report { overall_health: UNKNOWN } } }
I20260926 05:50:46.975023 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.972823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.11.247.3:32831)
0926 05:50:46.972887 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:46.972888 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:46.972897 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:50:46.972969 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:46.972970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:46.972985 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:46.973034 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:50:46.973147 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.973596 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.974282 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:46.974465 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:46.974517 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:46.974691 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:50:46.974816 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:50:46.974887 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:46.974937 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 05:50:46.974952 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:50:46.976735 32082 raft_consensus.cc:1261] T 8722ebb4c3f446659fcebcebd6adb2fc P ae69a4be7c1e4e9f83d1cea22a3c6632 [term 1 FOLLOWER]: Refusing update from remote peer 75e9acd03f31473cbf77cb463e38eb30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:46.976776 32213 raft_consensus.cc:1261] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 1 FOLLOWER]: Refusing update from remote peer 75e9acd03f31473cbf77cb463e38eb30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:50:46.977140 32414 consensus_queue.cc:1035] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae69a4be7c1e4e9f83d1cea22a3c6632" member_type: VOTER last_known_addr { host: "127.11.247.1" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:46.977258 32408 consensus_queue.cc:1035] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa22f0a779de439faaebb5201b57ee6a" member_type: VOTER last_known_addr { host: "127.11.247.2" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:50:46.980103 32418 mvcc.cc:205] Tried to move back new op lower bound from 7333485965214187520 to 7333485965179338752. Current Snapshot: MvccSnapshot[applied={T|T < 7333485965214187520}]
I20260926 05:50:46.980228 32419 mvcc.cc:205] Tried to move back new op lower bound from 7333485965214187520 to 7333485965179338752. Current Snapshot: MvccSnapshot[applied={T|T < 7333485965214187520}]
I20260926 05:50:46.981978 31939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:46.982077 31939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:50:46.982829 31939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f5f7ed8f58144bbbd743584a995a124: Sending DeleteTablet for 3 replicas of tablet 8722ebb4c3f446659fcebcebd6adb2fc
I20260926 05:50:46.983197 32193 tablet_service.cc:1514] Processing DeleteTablet for tablet 8722ebb4c3f446659fcebcebd6adb2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:46 UTC) from {username='slave'} at 127.0.0.1:54110
I20260926 05:50:46.983239 32324 tablet_service.cc:1514] Processing DeleteTablet for tablet 8722ebb4c3f446659fcebcebd6adb2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:46 UTC) from {username='slave'} at 127.0.0.1:44080
I20260926 05:50:46.983446 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 32000
I20260926 05:50:46.983436 32062 tablet_service.cc:1514] Processing DeleteTablet for tablet 8722ebb4c3f446659fcebcebd6adb2fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:50:46 UTC) from {username='slave'} at 127.0.0.1:51480
I20260926 05:50:46.985144 32428 tablet_replica.cc:330] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: stopping tablet replica
I20260926 05:50:46.985255 32428 raft_consensus.cc:2229] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:50:46.985385 32428 raft_consensus.cc:2258] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:46.985617 32427 tablet_replica.cc:330] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: stopping tablet replica
I20260926 05:50:46.985725 32427 raft_consensus.cc:2229] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:50:46.985841 32427 raft_consensus.cc:2258] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:50:46.986179 32428 ts_tablet_manager.cc:1905] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:46.986279 32427 ts_tablet_manager.cc:1905] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:50:46.988180 32428 ts_tablet_manager.cc:1918] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:46.988258 32427 ts_tablet_manager.cc:1918] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:50:46.988337 32427 log.cc:1196] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-1/wal/wals/8722ebb4c3f446659fcebcebd6adb2fc
I20260926 05:50:46.988353 32428 log.cc:1196] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Deleting WAL directory at /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401595249435-12252-0/minicluster-data/ts-2/wal/wals/8722ebb4c3f446659fcebcebd6adb2fc
I20260926 05:50:46.988601 32427 ts_tablet_manager.cc:1939] T 8722ebb4c3f446659fcebcebd6adb2fc P fa22f0a779de439faaebb5201b57ee6a: Deleting consensus metadata
I20260926 05:50:46.988832 32428 ts_tablet_manager.cc:1939] T 8722ebb4c3f446659fcebcebd6adb2fc P 75e9acd03f31473cbf77cb463e38eb30: Deleting consensus metadata
I20260926 05:50:46.989035 31925 catalog_manager.cc:4841] TS fa22f0a779de439faaebb5201b57ee6a (127.11.247.2:35317): tablet 8722ebb4c3f446659fcebcebd6adb2fc (table test-table [id=48cedaf74a1b41099369acc4b3ca2ea9]) successfully deleted
I20260926 05:50:46.989347 31924 catalog_manager.cc:4841] TS 75e9acd03f31473cbf77cb463e38eb30 (127.11.247.3:32831): tablet 8722ebb4c3f446659fcebcebd6adb2fc (table test-table [id=48cedaf74a1b41099369acc4b3ca2ea9]) successfully deleted
W20260926 05:50:46.990033 31924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.247.1:36093 (error 108)
I20260926 05:50:46.990545 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:46.990416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.1:36093 (local address 127.0.0.1:51492)
0926 05:50:46.990466 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:50:46.990506 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.1:36093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:50:46.990648 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 32131
W20260926 05:50:46.990664 31924 catalog_manager.cc:4584] TS ae69a4be7c1e4e9f83d1cea22a3c6632 (127.11.247.1:36093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8722ebb4c3f446659fcebcebd6adb2fc: Network error: Client connection negotiation failed: client connection to 127.11.247.1:36093: connect: Connection refused (error 111)
I20260926 05:50:46.995734 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 32262
I20260926 05:50:47.000550 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 31909
2026-09-26T05:50:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (720 ms)
[----------] 6 tests from EncryptionPolicyTest (8536 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:47 dist-test-slave-21p4 krb5kdc[32432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:47 dist-test-slave-21p4 krb5kdc[32432](info): set up 2 sockets
Sep 26 05:50:47 dist-test-slave-21p4 krb5kdc[32432](info): commencing operation
krb5kdc: starting...
W20260926 05:50:48.042737 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:48 dist-test-slave-21p4 krb5kdc[32432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:48.170668 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:48.255717 32445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:48.255964 32445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:48.255992 32445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:48.257325 32445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:48.257385 32445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:48.257400 32445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:48.257413 32445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:48.257426 32445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:48.259176 32445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:48.259492 32445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:48.259753 32445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:48.265692 32451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:48.265699 32450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:48.265699 32453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:48.266047 32445 server_base.cc:1034] running on GCE node
I20260926 05:50:48.266306 32445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:48.266686 32445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:48.267843 32445 hybrid_clock.cc:648] HybridClock initialized: now 1790401848267773 us; error 89 us; skew 500 ppm
Sep 26 05:50:48 dist-test-slave-21p4 krb5kdc[32432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:48.270932 32445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:48.271533 32445 webserver.cc:460] Webserver started at http://127.0.0.1:46663/ using document root <none> and password file <none>
I20260926 05:50:48.271824 32445 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:48.271916 32445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:48.272066 32445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:48.273684 32445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "eecdae9ab4934c3f904d12ddef9e0b6e"
format_stamp: "Formatted at 2026-09-26 05:50:48 on dist-test-slave-21p4"
server_key: "e66c2d7ef14e304c5a81e5e15497ffdf"
server_key_iv: "9aae536dcdf50fab52c4c6aedf4bfa54"
server_key_version: "encryptionkey@0"
I20260926 05:50:48.274233 32445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "eecdae9ab4934c3f904d12ddef9e0b6e"
format_stamp: "Formatted at 2026-09-26 05:50:48 on dist-test-slave-21p4"
server_key: "e66c2d7ef14e304c5a81e5e15497ffdf"
server_key_iv: "9aae536dcdf50fab52c4c6aedf4bfa54"
server_key_version: "encryptionkey@0"
I20260926 05:50:48.277478 32445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 05:50:48.280607 32460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:48.280857 32445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 05:50:48.281100 32445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "eecdae9ab4934c3f904d12ddef9e0b6e"
format_stamp: "Formatted at 2026-09-26 05:50:48 on dist-test-slave-21p4"
server_key: "e66c2d7ef14e304c5a81e5e15497ffdf"
server_key_iv: "9aae536dcdf50fab52c4c6aedf4bfa54"
server_key_version: "encryptionkey@0"
I20260926 05:50:48.281225 32445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:48.308269 32445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:48.312077 32445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:48.312291 32445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:48.317394 32445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33405
I20260926 05:50:48.317426 32512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33405 every 8 connection(s)
I20260926 05:50:48.317844 32445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:48.318993 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:48.321841 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e: Bootstrap starting.
I20260926 05:50:48.322468 32513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:48.322816 32513 log.cc:824] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:48.323608 32513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e: No bootstrap required, opened a new log
I20260926 05:50:48.325619 32513 raft_consensus.cc:348] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33405 } }
I20260926 05:50:48.325783 32513 raft_consensus.cc:374] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:48.325809 32513 raft_consensus.cc:729] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecdae9ab4934c3f904d12ddef9e0b6e, State: Initialized, Role: FOLLOWER
I20260926 05:50:48.325922 32513 consensus_queue.cc:260] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33405 } }
I20260926 05:50:48.326013 32513 raft_consensus.cc:388] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:48.326051 32513 raft_consensus.cc:482] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:48.326108 32513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:48.326683 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 32445
I20260926 05:50:48.326790 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:48.326889 32513 raft_consensus.cc:504] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33405 } }
I20260926 05:50:48.326982 12252 external_mini_cluster.cc:1428] Setting key cc460754db641a6670abcfcb7ebdd5f5
I20260926 05:50:48.327036 32513 leader_election.cc:302] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eecdae9ab4934c3f904d12ddef9e0b6e; no voters:
I20260926 05:50:48.327253 32513 leader_election.cc:288] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:48.327304 32516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:48.327469 32516 raft_consensus.cc:686] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [term 1 LEADER]: Becoming Leader. State: Replica: eecdae9ab4934c3f904d12ddef9e0b6e, State: Running, Role: LEADER
I20260926 05:50:48.327569 32516 consensus_queue.cc:237] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33405 } }
I20260926 05:50:48.327593 32513 sys_catalog.cc:564] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:48.327983 32516 sys_catalog.cc:455] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33405 } } }
I20260926 05:50:48.328058 32516 sys_catalog.cc:458] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:48.328176 32516 sys_catalog.cc:455] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eecdae9ab4934c3f904d12ddef9e0b6e. Latest consensus state: current_term: 1 leader_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecdae9ab4934c3f904d12ddef9e0b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33405 } } }
I20260926 05:50:48.328235 32516 sys_catalog.cc:458] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:48.328504 32524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:48.329051 32524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:48.331565 32524 catalog_manager.cc:1347] Generated new cluster ID: 9e8cfa2fe56f47419a1990e213bad40c
I20260926 05:50:48.331630 32524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:50:48 dist-test-slave-21p4 krb5kdc[32432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:50:48.342438 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:48.327567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48027 (local address 127.0.0.1:33405)
0926 05:50:48.327755 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:48.327765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:48.328510 (+ 745us) server_negotiation.cc:407] Connection header received
0926 05:50:48.328696 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:48.328705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:48.328794 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:48.328998 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:48.329964 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:48.331411 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:48.332496 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:48.332946 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:48.333092 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:48.335873 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:48.335879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:48.335879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:48.335894 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:48.338421 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:48.340432 (+ 2011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:48.340438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:48.340438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:48.340606 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:48.340903 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:48.340905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:48.340905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:48.341287 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:48.341483 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:48.342020 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 05:50:48.342155 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260926 05:50:48.344851 32524 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:48.345594 32524 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:48.346020 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:48.352773 32524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eecdae9ab4934c3f904d12ddef9e0b6e: Generated new TSK 0
I20260926 05:50:48.353036 32524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:48.360235 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:48.348149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52183 (local address 127.0.0.1:33405)
0926 05:50:48.348198 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:48.348201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:48.351042 (+ 2841us) server_negotiation.cc:407] Connection header received
0926 05:50:48.351143 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:48.351144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:48.351159 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:48.351235 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:48.352087 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:48.352700 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:48.353741 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:48.354100 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:48.354186 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:48.356038 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:48.356040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:48.356040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:48.356050 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:48.358118 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:48.358750 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:48.358752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:48.358752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:48.358833 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:48.359137 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:48.359138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:48.359139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:48.359328 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:48.359449 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:48.359996 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 05:50:48.360078 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:50:48.363272 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 32445
W20260926 05:50:48.370486 32430 connection.cc:535] client connection to 127.0.0.1:33405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33405): Connection reset by peer (error 104)
2026-09-26T05:50:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:48 dist-test-slave-21p4 krb5kdc[32544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:48 dist-test-slave-21p4 krb5kdc[32544](info): set up 2 sockets
Sep 26 05:50:48 dist-test-slave-21p4 krb5kdc[32544](info): commencing operation
krb5kdc: starting...
W20260926 05:50:49.402542 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:49 dist-test-slave-21p4 krb5kdc[32544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:49.521972 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:49.604943 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:49.605211 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:49.605253 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:49.606487 32557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:49.606565 32557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:49.606588 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:49.606642 32557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:49.606693 32557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:49.608294 32557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:49.608618 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:49.608857 32557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:49.613853 32565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:49.613842 32563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:49.613838 32562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:49.614174 32557 server_base.cc:1034] running on GCE node
I20260926 05:50:49.614499 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:49.614861 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:49.616060 32557 hybrid_clock.cc:648] HybridClock initialized: now 1790401849616026 us; error 64 us; skew 500 ppm
Sep 26 05:50:49 dist-test-slave-21p4 krb5kdc[32544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:49.618774 32557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:49.619289 32557 webserver.cc:460] Webserver started at http://127.0.0.1:42247/ using document root <none> and password file <none>
I20260926 05:50:49.619557 32557 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:49.619607 32557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:49.619747 32557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:49.620901 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "77e5d94d9bc04a64bc47aff8d611403d"
format_stamp: "Formatted at 2026-09-26 05:50:49 on dist-test-slave-21p4"
server_key: "40a2aeb6d06a5c7bdba10290df41b57c"
server_key_iv: "26220630c2cab0b5ffae766708c49f66"
server_key_version: "encryptionkey@0"
I20260926 05:50:49.621282 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "77e5d94d9bc04a64bc47aff8d611403d"
format_stamp: "Formatted at 2026-09-26 05:50:49 on dist-test-slave-21p4"
server_key: "40a2aeb6d06a5c7bdba10290df41b57c"
server_key_iv: "26220630c2cab0b5ffae766708c49f66"
server_key_version: "encryptionkey@0"
I20260926 05:50:49.622745 32557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:50:49.623620 32572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:49.623807 32557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:49.623903 32557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "77e5d94d9bc04a64bc47aff8d611403d"
format_stamp: "Formatted at 2026-09-26 05:50:49 on dist-test-slave-21p4"
server_key: "40a2aeb6d06a5c7bdba10290df41b57c"
server_key_iv: "26220630c2cab0b5ffae766708c49f66"
server_key_version: "encryptionkey@0"
I20260926 05:50:49.623980 32557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:49.667642 32557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:49.670015 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:49.670128 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:49.674170 32557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46077
I20260926 05:50:49.674206 32624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46077 every 8 connection(s)
I20260926 05:50:49.674567 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:49.675545 32625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:49.677214 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 32557
I20260926 05:50:49.677340 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:49.677532 12252 external_mini_cluster.cc:1428] Setting key 6a88849cfa407651f18b28baf56b9f56
I20260926 05:50:49.678145 32625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d: Bootstrap starting.
I20260926 05:50:49.678841 32625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:49.679155 32625 log.cc:824] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:49.679905 32625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d: No bootstrap required, opened a new log
I20260926 05:50:49.682116 32625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e5d94d9bc04a64bc47aff8d611403d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260926 05:50:49.682313 32625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:49.682358 32625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e5d94d9bc04a64bc47aff8d611403d, State: Initialized, Role: FOLLOWER
I20260926 05:50:49.682490 32625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e5d94d9bc04a64bc47aff8d611403d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260926 05:50:49.682556 32625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:49.682629 32625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:49.682683 32625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:49.683418 32625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e5d94d9bc04a64bc47aff8d611403d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260926 05:50:49.683560 32625 leader_election.cc:302] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77e5d94d9bc04a64bc47aff8d611403d; no voters:
I20260926 05:50:49.683758 32625 leader_election.cc:288] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:49.683791 32630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:49.683843 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:49.678134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36555 (local address 127.0.0.1:46077)
0926 05:50:49.678337 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:49.678349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:49.678378 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:50:49.678974 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:49.678982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:49.679032 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:49.679152 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:49.679436 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:49.680502 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:49.681593 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:49.682045 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:49.682131 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:49.683041 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:49.683045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:49.683045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:49.683209 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:49.683241 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:49.683251 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:49.683333 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:49.683459 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:50:49.683568 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260926 05:50:49.683917 32630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [term 1 LEADER]: Becoming Leader. State: Replica: 77e5d94d9bc04a64bc47aff8d611403d, State: Running, Role: LEADER
I20260926 05:50:49.683992 32630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e5d94d9bc04a64bc47aff8d611403d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260926 05:50:49.684024 32625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:49.684320 32630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77e5d94d9bc04a64bc47aff8d611403d. Latest consensus state: current_term: 1 leader_uuid: "77e5d94d9bc04a64bc47aff8d611403d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e5d94d9bc04a64bc47aff8d611403d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } } }
I20260926 05:50:49.684393 32630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:49.684345 32631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77e5d94d9bc04a64bc47aff8d611403d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e5d94d9bc04a64bc47aff8d611403d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } } }
I20260926 05:50:49.684551 32631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:49.684787 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 05:50:49.685886 32646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:50:49.685948 32646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:50:49.686291 32635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:49.686522 32635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:49.688140 32635 catalog_manager.cc:1347] Generated new cluster ID: dcf70418ed474ef4a447de438bf95cc6
I20260926 05:50:49.688189 32635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:49.690256 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:49.686401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37983 (local address 127.0.0.1:46077)
0926 05:50:49.686436 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:49.686438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:49.686626 (+ 188us) server_negotiation.cc:407] Connection header received
0926 05:50:49.686731 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:49.686733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:49.686743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:49.686805 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:49.686960 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:49.687679 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:49.688842 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:49.689151 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:49.689196 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:49.689983 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:49.689984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:49.689984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:49.689989 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:49.690002 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:49.690007 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:49.690039 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:49.690170 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 05:50:49.690183 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:50:49.702230 32635 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:49.703080 32635 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:49.715019 32635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77e5d94d9bc04a64bc47aff8d611403d: Generated new TSK 0
I20260926 05:50:49.715317 32635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:49.728010 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 32557
W20260926 05:50:49.732404 32542 connection.cc:535] client connection to 127.0.0.1:46077 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46077): Connection reset by peer (error 104)
2026-09-26T05:50:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:49 dist-test-slave-21p4 krb5kdc[32656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:49 dist-test-slave-21p4 krb5kdc[32656](info): set up 2 sockets
Sep 26 05:50:49 dist-test-slave-21p4 krb5kdc[32656](info): commencing operation
krb5kdc: starting...
W20260926 05:50:50.762750 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:50 dist-test-slave-21p4 krb5kdc[32656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:50.885097 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:50.966776 32669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:50.966987 32669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:50.967010 32669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:50.968093 32669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:50.968132 32669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:50.968146 32669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:50.968159 32669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:50.968171 32669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:50.969677 32669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:50.969889 32669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:50.970064 32669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:50.974886 32675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:50.974898 32674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:50.974898 32677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:50.975334 32669 server_base.cc:1034] running on GCE node
I20260926 05:50:50.975613 32669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:50.975912 32669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:50.977068 32669 hybrid_clock.cc:648] HybridClock initialized: now 1790401850977051 us; error 25 us; skew 500 ppm
Sep 26 05:50:50 dist-test-slave-21p4 krb5kdc[32656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:50.987001 32669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:50.987524 32669 webserver.cc:460] Webserver started at http://127.0.0.1:40067/ using document root <none> and password file <none>
I20260926 05:50:50.987759 32669 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:50.987831 32669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:50.987988 32669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:50.989123 32669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "25032ab41dc24e9fb6cfd5470e58e959"
format_stamp: "Formatted at 2026-09-26 05:50:50 on dist-test-slave-21p4"
server_key: "071ac43ced4ea3034830993a504ddfdf"
server_key_iv: "041f214c9d7065a9cff152c794abb0a2"
server_key_version: "encryptionkey@0"
I20260926 05:50:50.989494 32669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "25032ab41dc24e9fb6cfd5470e58e959"
format_stamp: "Formatted at 2026-09-26 05:50:50 on dist-test-slave-21p4"
server_key: "071ac43ced4ea3034830993a504ddfdf"
server_key_iv: "041f214c9d7065a9cff152c794abb0a2"
server_key_version: "encryptionkey@0"
I20260926 05:50:50.991005 32669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:50:50.991832 32684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:50.992050 32669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:50.992154 32669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "25032ab41dc24e9fb6cfd5470e58e959"
format_stamp: "Formatted at 2026-09-26 05:50:50 on dist-test-slave-21p4"
server_key: "071ac43ced4ea3034830993a504ddfdf"
server_key_iv: "041f214c9d7065a9cff152c794abb0a2"
server_key_version: "encryptionkey@0"
I20260926 05:50:50.992242 32669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:50.996003 32669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:50.998703 32669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:50.998862 32669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:51.002969 32669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37659
I20260926 05:50:51.003000 32736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37659 every 8 connection(s)
I20260926 05:50:51.003345 32669 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:51.004269 32737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:51.006685 32737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959: Bootstrap starting.
I20260926 05:50:51.007279 32737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:51.007576 32737 log.cc:824] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:51.008281 32737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959: No bootstrap required, opened a new log
I20260926 05:50:51.009913 32737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25032ab41dc24e9fb6cfd5470e58e959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260926 05:50:51.010082 32737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:51.010130 32737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25032ab41dc24e9fb6cfd5470e58e959, State: Initialized, Role: FOLLOWER
I20260926 05:50:51.010288 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 32669
I20260926 05:50:51.010304 32737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25032ab41dc24e9fb6cfd5470e58e959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260926 05:50:51.010402 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:51.010402 32737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:51.010442 32737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:51.010491 32737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:51.010660 12252 external_mini_cluster.cc:1428] Setting key 2d30ee16c7648929621ab3107a67f5f5
I20260926 05:50:51.011173 32737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25032ab41dc24e9fb6cfd5470e58e959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260926 05:50:51.011291 32737 leader_election.cc:302] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25032ab41dc24e9fb6cfd5470e58e959; no voters:
I20260926 05:50:51.011453 32737 leader_election.cc:288] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:51.011482 32742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:51.011657 32742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [term 1 LEADER]: Becoming Leader. State: Replica: 25032ab41dc24e9fb6cfd5470e58e959, State: Running, Role: LEADER
I20260926 05:50:51.011706 32737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:51.011780 32742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25032ab41dc24e9fb6cfd5470e58e959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } }
I20260926 05:50:51.012450 32744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25032ab41dc24e9fb6cfd5470e58e959. Latest consensus state: current_term: 1 leader_uuid: "25032ab41dc24e9fb6cfd5470e58e959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25032ab41dc24e9fb6cfd5470e58e959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } } }
I20260926 05:50:51.012486 32743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25032ab41dc24e9fb6cfd5470e58e959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25032ab41dc24e9fb6cfd5470e58e959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37659 } } }
I20260926 05:50:51.012619 32744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:51.012635 32743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:51.013182 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:51.011171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38703 (local address 127.0.0.1:37659)
0926 05:50:51.011361 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:51.011371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:51.011409 (+ 38us) server_negotiation.cc:407] Connection header received
0926 05:50:51.011540 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:51.011548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:51.011582 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:51.012056 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:51.012558 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:51.012562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:51.012562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:51.012567 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:51.012614 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:51.012628 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:51.012905 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:51.012938 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 05:50:51.012978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260926 05:50:51.013913 32757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:50:51.014014 32757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:50:51.014194 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:51.014793 32751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:51.014976 32751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:51.016444 32751 catalog_manager.cc:1347] Generated new cluster ID: dcc230bbabff477fb915784c48aa4d3a
I20260926 05:50:51.016497 32751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:51.023914 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:51.015747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46323 (local address 127.0.0.1:37659)
0926 05:50:51.015941 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:51.015943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:51.022850 (+ 6907us) server_negotiation.cc:407] Connection header received
0926 05:50:51.022927 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:51.022928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:51.022950 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:51.023020 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:51.023693 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:51.023694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:51.023694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:51.023698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:51.023712 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:51.023716 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:51.023771 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:51.023824 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:50:51.023837 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260926 05:50:51.028095 32751 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:51.028293 32751 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:51.038770 32751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25032ab41dc24e9fb6cfd5470e58e959: Generated new TSK 0
I20260926 05:50:51.039072 32751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:51.048041 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 32669
2026-09-26T05:50:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:51 dist-test-slave-21p4 krb5kdc[300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:51 dist-test-slave-21p4 krb5kdc[300](info): set up 2 sockets
Sep 26 05:50:51 dist-test-slave-21p4 krb5kdc[300](info): commencing operation
krb5kdc: starting...
W20260926 05:50:52.094868 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:52 dist-test-slave-21p4 krb5kdc[300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:52.214088 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:52.298220 313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:52.298486 313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:52.298532 313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:52.299775 313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:52.299837 313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:52.299851 313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:52.299865 313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:52.299877 313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:52.301334 313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:52.301544 313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:52.301726 313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:52.306761 318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:52.306761 321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:52.306797 319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:52.306921 313 server_base.cc:1034] running on GCE node
I20260926 05:50:52.307267 313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:52.307542 313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:52.308709 313 hybrid_clock.cc:648] HybridClock initialized: now 1790401852308691 us; error 28 us; skew 500 ppm
Sep 26 05:50:52 dist-test-slave-21p4 krb5kdc[300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:52.311522 313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:52.312021 313 webserver.cc:460] Webserver started at http://127.0.0.1:45111/ using document root <none> and password file <none>
I20260926 05:50:52.312217 313 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:52.312263 313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:52.312355 313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:52.313417 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "333a65767c6e4f2185c07b3f64a83e37"
format_stamp: "Formatted at 2026-09-26 05:50:52 on dist-test-slave-21p4"
server_key: "4d9516ac450601b171ac96d3eb018063"
server_key_iv: "e8b6e0f107ba3af1e9cabd30a6be88c0"
server_key_version: "encryptionkey@0"
I20260926 05:50:52.313740 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "333a65767c6e4f2185c07b3f64a83e37"
format_stamp: "Formatted at 2026-09-26 05:50:52 on dist-test-slave-21p4"
server_key: "4d9516ac450601b171ac96d3eb018063"
server_key_iv: "e8b6e0f107ba3af1e9cabd30a6be88c0"
server_key_version: "encryptionkey@0"
I20260926 05:50:52.315274 313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:50:52.316082 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:52.316294 313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:52.316412 313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "333a65767c6e4f2185c07b3f64a83e37"
format_stamp: "Formatted at 2026-09-26 05:50:52 on dist-test-slave-21p4"
server_key: "4d9516ac450601b171ac96d3eb018063"
server_key_iv: "e8b6e0f107ba3af1e9cabd30a6be88c0"
server_key_version: "encryptionkey@0"
I20260926 05:50:52.316498 313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:52.328970 313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:52.331478 313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:52.331660 313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:52.335990 313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37907
I20260926 05:50:52.336035 380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37907 every 8 connection(s)
I20260926 05:50:52.336408 313 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:52.337338 381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:52.339286 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 313
I20260926 05:50:52.339434 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:52.339617 12252 external_mini_cluster.cc:1428] Setting key 67bf3c866f2c2b9b5b86bcf9c12baa49
I20260926 05:50:52.340021 381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37: Bootstrap starting.
I20260926 05:50:52.341058 381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:52.341409 381 log.cc:824] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:52.342231 381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37: No bootstrap required, opened a new log
I20260926 05:50:52.344632 381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333a65767c6e4f2185c07b3f64a83e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 05:50:52.344821 381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:52.344856 381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333a65767c6e4f2185c07b3f64a83e37, State: Initialized, Role: FOLLOWER
I20260926 05:50:52.344981 381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333a65767c6e4f2185c07b3f64a83e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 05:50:52.345059 381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:52.345096 381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:52.345152 381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:52.345948 381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333a65767c6e4f2185c07b3f64a83e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 05:50:52.346046 381 leader_election.cc:302] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333a65767c6e4f2185c07b3f64a83e37; no voters:
I20260926 05:50:52.346302 381 leader_election.cc:288] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:52.346357 386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:52.346530 386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [term 1 LEADER]: Becoming Leader. State: Replica: 333a65767c6e4f2185c07b3f64a83e37, State: Running, Role: LEADER
I20260926 05:50:52.346630 381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:52.346753 386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333a65767c6e4f2185c07b3f64a83e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260926 05:50:52.347123 387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333a65767c6e4f2185c07b3f64a83e37. Latest consensus state: current_term: 1 leader_uuid: "333a65767c6e4f2185c07b3f64a83e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333a65767c6e4f2185c07b3f64a83e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } } }
I20260926 05:50:52.347134 388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333a65767c6e4f2185c07b3f64a83e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333a65767c6e4f2185c07b3f64a83e37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } } }
I20260926 05:50:52.347276 387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:52.347517 388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37 [sys.catalog]: This master's current role is: LEADER
Sep 26 05:50:52 dist-test-slave-21p4 krb5kdc[300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:50:52.347741 393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:52.348678 393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:52.351105 393 catalog_manager.cc:1347] Generated new cluster ID: 51ca47a0db5a4e2b8aaea893dcb86f26
I20260926 05:50:52.351159 393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:52.352274 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:52.340220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43881 (local address 127.0.0.1:37907)
0926 05:50:52.340415 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:52.340425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:52.340459 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:50:52.341103 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:52.341114 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:52.341185 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:52.341352 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:52.342206 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:52.343661 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:52.344435 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:52.345067 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:52.345185 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:52.348313 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:52.348318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:52.348318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:52.348334 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:52.350439 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:52.350970 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:52.350974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:52.350974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:52.351092 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:52.351321 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:52.351322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:52.351322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:52.351580 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:52.351723 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:52.351856 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 05:50:52.352035 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260926 05:50:52.353195 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:52.365562 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:52.354567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36323 (local address 127.0.0.1:37907)
0926 05:50:52.354638 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:52.354640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:52.359104 (+ 4464us) server_negotiation.cc:407] Connection header received
0926 05:50:52.359193 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:52.359194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:52.359205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:52.359256 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:52.359894 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:52.360396 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:52.361135 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:52.361378 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:52.361460 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:52.362795 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:52.362796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:52.362796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:52.362804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:52.364228 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:52.364605 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:52.364607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:52.364608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:52.364694 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:52.364942 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:52.364943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:52.364943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:52.365073 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:52.365188 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:52.365382 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 05:50:52.365466 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:50:52.365635 393 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:52.366396 393 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:52.380350 393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333a65767c6e4f2185c07b3f64a83e37: Generated new TSK 0
I20260926 05:50:52.380586 393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:52.385742 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 313
2026-09-26T05:50:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:52 dist-test-slave-21p4 krb5kdc[412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:52 dist-test-slave-21p4 krb5kdc[412](info): set up 2 sockets
Sep 26 05:50:52 dist-test-slave-21p4 krb5kdc[412](info): commencing operation
krb5kdc: starting...
W20260926 05:50:53.430797 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:53 dist-test-slave-21p4 krb5kdc[412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:53.557530 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:53.642769 425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:53.643024 425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:53.643064 425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:53.644282 425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:53.644352 425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:53.644378 425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:53.644407 425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:53.644435 425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:53.645994 425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:53.646292 425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:53.646538 425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:53.651472 431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:53.651472 430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:53.651500 433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:53.651710 425 server_base.cc:1034] running on GCE node
I20260926 05:50:53.651975 425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:53.652285 425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:53.653481 425 hybrid_clock.cc:648] HybridClock initialized: now 1790401853653458 us; error 42 us; skew 500 ppm
Sep 26 05:50:53 dist-test-slave-21p4 krb5kdc[412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:53.656107 425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:53.656589 425 webserver.cc:460] Webserver started at http://127.0.0.1:38537/ using document root <none> and password file <none>
I20260926 05:50:53.656782 425 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:53.656824 425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:53.656915 425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:53.658294 425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "3ed2b93bebfc4e8abf7eb8900051ff63"
format_stamp: "Formatted at 2026-09-26 05:50:53 on dist-test-slave-21p4"
server_key: "cc83289d97533e3499b773cbac8fd368"
server_key_iv: "d5ca9762621d75052630a64761a51b2f"
server_key_version: "encryptionkey@0"
I20260926 05:50:53.658707 425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "3ed2b93bebfc4e8abf7eb8900051ff63"
format_stamp: "Formatted at 2026-09-26 05:50:53 on dist-test-slave-21p4"
server_key: "cc83289d97533e3499b773cbac8fd368"
server_key_iv: "d5ca9762621d75052630a64761a51b2f"
server_key_version: "encryptionkey@0"
I20260926 05:50:53.660099 425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:53.660853 440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:53.661067 425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:50:53.661162 425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "3ed2b93bebfc4e8abf7eb8900051ff63"
format_stamp: "Formatted at 2026-09-26 05:50:53 on dist-test-slave-21p4"
server_key: "cc83289d97533e3499b773cbac8fd368"
server_key_iv: "d5ca9762621d75052630a64761a51b2f"
server_key_version: "encryptionkey@0"
I20260926 05:50:53.661222 425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:53.671145 425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:53.673275 425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:53.673390 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:53.677292 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40227
I20260926 05:50:53.677385 492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40227 every 8 connection(s)
I20260926 05:50:53.677727 425 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:53.678804 493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:53.681341 493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63: Bootstrap starting.
I20260926 05:50:53.681949 493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:53.682243 493 log.cc:824] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:53.682721 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 425
I20260926 05:50:53.682924 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:53.683138 12252 external_mini_cluster.cc:1428] Setting key e6a902b7bd79141eb39d59e186a5f942
I20260926 05:50:53.683141 493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63: No bootstrap required, opened a new log
I20260926 05:50:53.685550 493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260926 05:50:53.685711 493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:53.685776 493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed2b93bebfc4e8abf7eb8900051ff63, State: Initialized, Role: FOLLOWER
I20260926 05:50:53.685957 493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260926 05:50:53.686057 493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:53.686105 493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:53.686157 493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:53.686817 493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260926 05:50:53.686950 493 leader_election.cc:302] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ed2b93bebfc4e8abf7eb8900051ff63; no voters:
I20260926 05:50:53.687151 493 leader_election.cc:288] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:53.687180 498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:53.687381 498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed2b93bebfc4e8abf7eb8900051ff63, State: Running, Role: LEADER
I20260926 05:50:53.687510 493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:53.687886 498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260926 05:50:53.688550 500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260926 05:50:53.688719 500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:53.688953 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:53.683667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45453 (local address 127.0.0.1:40227)
0926 05:50:53.683875 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:53.683884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:53.683910 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:50:53.684414 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:53.684420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:53.684467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:53.684615 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:53.684796 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:53.685651 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:53.686658 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:53.687148 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:53.687267 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:53.687982 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:53.687986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:53.687986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:53.688201 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:53.688243 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:53.688257 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:53.688609 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:53.688645 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 05:50:53.688712 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260926 05:50:53.689044 499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ed2b93bebfc4e8abf7eb8900051ff63. Latest consensus state: current_term: 1 leader_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ed2b93bebfc4e8abf7eb8900051ff63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260926 05:50:53.689318 499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63 [sys.catalog]: This master's current role is: LEADER
W20260926 05:50:53.689456 512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:50:53.689517 512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:50:53.689582 514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:53.689757 514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:53.690191 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:53.691275 514 catalog_manager.cc:1347] Generated new cluster ID: 2dc644c354234850a0e29a8ef2ad3e47
I20260926 05:50:53.691326 514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:53.697983 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:53.691701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56073 (local address 127.0.0.1:40227)
0926 05:50:53.694680 (+ 2979us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:53.694682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:53.694690 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:50:53.694707 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:53.694708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:53.694718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:53.694769 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:53.694935 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:53.695652 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:53.696636 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:53.696952 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:53.697028 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:53.697735 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:53.697737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:53.697737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:53.697742 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:53.697759 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:53.697765 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:53.697825 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:53.697880 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:50:53.697901 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2958}
I20260926 05:50:53.707757 514 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:53.708302 514 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:53.714404 514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ed2b93bebfc4e8abf7eb8900051ff63: Generated new TSK 0
I20260926 05:50:53.714645 514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:53.717163 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 425
2026-09-26T05:50:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:53 dist-test-slave-21p4 krb5kdc[524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:53 dist-test-slave-21p4 krb5kdc[524](info): set up 2 sockets
Sep 26 05:50:53 dist-test-slave-21p4 krb5kdc[524](info): commencing operation
krb5kdc: starting...
W20260926 05:50:54.751029 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:54 dist-test-slave-21p4 krb5kdc[524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:54.870054 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:54.951924 537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:54.952154 537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:54.952181 537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:54.953423 537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:54.953474 537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:54.953488 537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:54.953502 537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:54.953514 537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:54.955092 537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:54.955374 537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:54.955628 537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:54.960737 542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:54.960808 537 server_base.cc:1034] running on GCE node
W20260926 05:50:54.960754 545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:54.960737 543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:54.961253 537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:54.961601 537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:54.962765 537 hybrid_clock.cc:648] HybridClock initialized: now 1790401854962751 us; error 47 us; skew 500 ppm
Sep 26 05:50:54 dist-test-slave-21p4 krb5kdc[524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:54.965310 537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:54.965831 537 webserver.cc:460] Webserver started at http://127.0.0.1:40183/ using document root <none> and password file <none>
I20260926 05:50:54.966090 537 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:54.966162 537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:54.966298 537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:54.967455 537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "4e435c6763e6475f80cb3349e236528a"
format_stamp: "Formatted at 2026-09-26 05:50:54 on dist-test-slave-21p4"
server_key: "8b01b8342d47e9ffa5ba102aec64a23c"
server_key_iv: "2c7f4f6f2cf032b218f0ab72208a5ae5"
server_key_version: "encryptionkey@0"
I20260926 05:50:54.967825 537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "4e435c6763e6475f80cb3349e236528a"
format_stamp: "Formatted at 2026-09-26 05:50:54 on dist-test-slave-21p4"
server_key: "8b01b8342d47e9ffa5ba102aec64a23c"
server_key_iv: "2c7f4f6f2cf032b218f0ab72208a5ae5"
server_key_version: "encryptionkey@0"
I20260926 05:50:54.969257 537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:50:54.970069 552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:54.970260 537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:54.970359 537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "4e435c6763e6475f80cb3349e236528a"
format_stamp: "Formatted at 2026-09-26 05:50:54 on dist-test-slave-21p4"
server_key: "8b01b8342d47e9ffa5ba102aec64a23c"
server_key_iv: "2c7f4f6f2cf032b218f0ab72208a5ae5"
server_key_version: "encryptionkey@0"
I20260926 05:50:54.970463 537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:54.973963 537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:54.976119 537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:54.976225 537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:54.980096 537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38185
I20260926 05:50:54.980150 604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38185 every 8 connection(s)
I20260926 05:50:54.980475 537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:54.981388 605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:54.983779 605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a: Bootstrap starting.
I20260926 05:50:54.984373 605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:54.984653 605 log.cc:824] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:54.985277 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 537
I20260926 05:50:54.985407 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:54.985478 605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a: No bootstrap required, opened a new log
I20260926 05:50:54.985607 12252 external_mini_cluster.cc:1428] Setting key a12b921e076dc3d58f903a00c64e8816
I20260926 05:50:54.987453 605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e435c6763e6475f80cb3349e236528a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260926 05:50:54.987618 605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:54.987680 605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e435c6763e6475f80cb3349e236528a, State: Initialized, Role: FOLLOWER
I20260926 05:50:54.987840 605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e435c6763e6475f80cb3349e236528a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260926 05:50:54.988214 605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:54.988297 605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:54.988363 605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:54.988874 609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:54.986178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43593 (local address 127.0.0.1:38185)
0926 05:50:54.986385 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:54.986396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:54.986439 (+ 43us) server_negotiation.cc:407] Connection header received
0926 05:50:54.987443 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:54.987451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:54.987483 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:54.987660 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:54.988380 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:54.988385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:54.988385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:54.988389 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:54.988443 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:54.988474 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:54.988536 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:54.988606 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 05:50:54.988650 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260926 05:50:54.989650 605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e435c6763e6475f80cb3349e236528a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260926 05:50:54.989763 605 leader_election.cc:302] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e435c6763e6475f80cb3349e236528a; no voters:
I20260926 05:50:54.989930 605 leader_election.cc:288] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:54.989960 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:54.989979 610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:54.990154 610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [term 1 LEADER]: Becoming Leader. State: Replica: 4e435c6763e6475f80cb3349e236528a, State: Running, Role: LEADER
I20260926 05:50:54.990185 605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:54.991559 610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e435c6763e6475f80cb3349e236528a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260926 05:50:54.992260 611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e435c6763e6475f80cb3349e236528a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e435c6763e6475f80cb3349e236528a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } } }
I20260926 05:50:54.992372 611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:54.992740 629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:54.992898 609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:54.991564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42173 (local address 127.0.0.1:38185)
0926 05:50:54.991606 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:54.991608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:54.991728 (+ 120us) server_negotiation.cc:407] Connection header received
0926 05:50:54.991833 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:54.991834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:54.991840 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:54.991890 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:54.992584 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:54.992586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:54.992587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:54.992592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:54.992612 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:54.992617 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:54.992699 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:54.992803 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 05:50:54.992821 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:50:54.992980 612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e435c6763e6475f80cb3349e236528a. Latest consensus state: current_term: 1 leader_uuid: "4e435c6763e6475f80cb3349e236528a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e435c6763e6475f80cb3349e236528a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } } }
I20260926 05:50:54.993053 612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:54.993443 629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:54.995033 629 catalog_manager.cc:1347] Generated new cluster ID: 4de173cf9f1b4f5ab04dd2dfd3b0dc2c
I20260926 05:50:54.995091 629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:55.008746 629 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:55.008971 629 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:55.015368 629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e435c6763e6475f80cb3349e236528a: Generated new TSK 0
I20260926 05:50:55.015596 629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:55.021982 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 537
2026-09-26T05:50:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:55 dist-test-slave-21p4 krb5kdc[636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:55 dist-test-slave-21p4 krb5kdc[636](info): set up 2 sockets
Sep 26 05:50:55 dist-test-slave-21p4 krb5kdc[636](info): commencing operation
krb5kdc: starting...
W20260926 05:50:56.074810 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:56 dist-test-slave-21p4 krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:56.198693 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:56.283838 649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:56.284070 649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:56.284093 649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:56.285282 649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:56.285332 649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:56.285346 649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:56.285359 649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:56.285372 649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:56.286963 649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:56.287189 649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:56.287365 649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:56.292382 654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:56.292333 657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:56.292457 649 server_base.cc:1034] running on GCE node
W20260926 05:50:56.292333 655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:56.292892 649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:56.293224 649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:56.294425 649 hybrid_clock.cc:648] HybridClock initialized: now 1790401856294390 us; error 51 us; skew 500 ppm
Sep 26 05:50:56 dist-test-slave-21p4 krb5kdc[636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:56.301420 649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:56.301947 649 webserver.cc:460] Webserver started at http://127.0.0.1:44043/ using document root <none> and password file <none>
I20260926 05:50:56.302201 649 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:56.302273 649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:56.302428 649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:56.303666 649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "bca2623f52fc4d869290b5f632496d9c"
format_stamp: "Formatted at 2026-09-26 05:50:56 on dist-test-slave-21p4"
server_key: "fcbcfd40e79c79597cad3c87c872ab35"
server_key_iv: "184ef1cebf87c254d5fcd0eaeea855cf"
server_key_version: "encryptionkey@0"
I20260926 05:50:56.304044 649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "bca2623f52fc4d869290b5f632496d9c"
format_stamp: "Formatted at 2026-09-26 05:50:56 on dist-test-slave-21p4"
server_key: "fcbcfd40e79c79597cad3c87c872ab35"
server_key_iv: "184ef1cebf87c254d5fcd0eaeea855cf"
server_key_version: "encryptionkey@0"
I20260926 05:50:56.305406 649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:50:56.306190 664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:56.306370 649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:50:56.306471 649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "bca2623f52fc4d869290b5f632496d9c"
format_stamp: "Formatted at 2026-09-26 05:50:56 on dist-test-slave-21p4"
server_key: "fcbcfd40e79c79597cad3c87c872ab35"
server_key_iv: "184ef1cebf87c254d5fcd0eaeea855cf"
server_key_version: "encryptionkey@0"
I20260926 05:50:56.306559 649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:56.333124 649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:56.335582 649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:56.335750 649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:56.339905 649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33679
I20260926 05:50:56.339942 716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33679 every 8 connection(s)
I20260926 05:50:56.340334 649 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:56.341233 717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:56.343580 717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c: Bootstrap starting.
I20260926 05:50:56.344002 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 649
I20260926 05:50:56.344162 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:56.344247 717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:56.344412 12252 external_mini_cluster.cc:1428] Setting key d696d76acdb65373568716ade258811f
I20260926 05:50:56.344586 717 log.cc:824] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:56.345376 717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c: No bootstrap required, opened a new log
I20260926 05:50:56.347728 717 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca2623f52fc4d869290b5f632496d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33679 } }
I20260926 05:50:56.347911 717 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:56.347949 717 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca2623f52fc4d869290b5f632496d9c, State: Initialized, Role: FOLLOWER
I20260926 05:50:56.348069 717 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca2623f52fc4d869290b5f632496d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33679 } }
I20260926 05:50:56.348143 717 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:56.348172 717 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:56.348214 717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:56.348995 717 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca2623f52fc4d869290b5f632496d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33679 } }
I20260926 05:50:56.349128 717 leader_election.cc:302] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca2623f52fc4d869290b5f632496d9c; no voters:
I20260926 05:50:56.349332 717 leader_election.cc:288] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:56.349373 722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:56.349625 722 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [term 1 LEADER]: Becoming Leader. State: Replica: bca2623f52fc4d869290b5f632496d9c, State: Running, Role: LEADER
I20260926 05:50:56.349797 722 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca2623f52fc4d869290b5f632496d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33679 } }
I20260926 05:50:56.349692 717 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 05:50:56 dist-test-slave-21p4 krb5kdc[636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:50:56.351585 723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca2623f52fc4d869290b5f632496d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca2623f52fc4d869290b5f632496d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33679 } } }
I20260926 05:50:56.352388 723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:56.351716 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca2623f52fc4d869290b5f632496d9c. Latest consensus state: current_term: 1 leader_uuid: "bca2623f52fc4d869290b5f632496d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca2623f52fc4d869290b5f632496d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33679 } } }
I20260926 05:50:56.352582 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c [sys.catalog]: This master's current role is: LEADER
W20260926 05:50:56.351788 737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:50:56.352699 737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:50:56.352785 739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:56.353055 739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:56.354785 739 catalog_manager.cc:1347] Generated new cluster ID: f8b5b843da3e4381b862936e3e478753
I20260926 05:50:56.354837 739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:56.356544 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:56.344914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:52867 (local address 127.0.0.1:33679)
0926 05:50:56.345104 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:56.345113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:56.345147 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:50:56.345651 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:56.345658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:56.345709 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:56.345825 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:56.346683 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:56.347564 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:56.348742 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:56.349189 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:56.349301 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:56.352362 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:56.352366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:56.352366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:56.352379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:56.354359 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:56.354873 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:56.354876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:56.354876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:56.354987 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:56.355236 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:56.355237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:56.355237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:56.355566 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:56.355724 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:56.356217 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 05:50:56.356321 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260926 05:50:56.357615 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:56.366657 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:56.359044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:58541 (local address 127.0.0.1:33679)
0926 05:50:56.359079 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:56.359081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:56.359238 (+ 157us) server_negotiation.cc:407] Connection header received
0926 05:50:56.359279 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:56.359280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:56.359291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:56.359344 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:56.359921 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:56.360857 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:56.361834 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:56.362175 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:56.362250 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:56.363695 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:56.363696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:56.363697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:50:56.363704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:56.365318 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:56.365729 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:56.365730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:56.365730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:56.365792 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:50:56.366014 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:50:56.366015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:50:56.366015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:50:56.366152 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:56.366268 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:56.366449 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 05:50:56.366518 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:50:56.370333 739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:56.371083 739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:56.376021 739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca2623f52fc4d869290b5f632496d9c: Generated new TSK 0
I20260926 05:50:56.376216 739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:56.384933 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 649
W20260926 05:50:56.390197 634 connection.cc:535] client connection to 127.0.0.1:33679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33679): Connection reset by peer (error 104)
2026-09-26T05:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:56 dist-test-slave-21p4 krb5kdc[748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:56 dist-test-slave-21p4 krb5kdc[748](info): set up 2 sockets
Sep 26 05:50:56 dist-test-slave-21p4 krb5kdc[748](info): commencing operation
krb5kdc: starting...
W20260926 05:50:57.438910 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:57 dist-test-slave-21p4 krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:57.559455 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:57.644471 761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:57.644678 761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:57.644701 761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:57.645851 761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:57.645895 761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:57.645908 761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:57.645920 761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:57.645932 761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:57.647349 761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:57.647536 761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:57.647707 761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:50:57.652627 761 server_base.cc:1034] running on GCE node
W20260926 05:50:57.652599 767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:57.652619 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:57.652599 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:57.653124 761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:57.653434 761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:57.654642 761 hybrid_clock.cc:648] HybridClock initialized: now 1790401857654612 us; error 65 us; skew 500 ppm
Sep 26 05:50:57 dist-test-slave-21p4 krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:57.657073 761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:57.657498 761 webserver.cc:460] Webserver started at http://127.0.0.1:44917/ using document root <none> and password file <none>
I20260926 05:50:57.657729 761 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:57.657799 761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:57.657958 761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:57.659165 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "8948dae1c660429492a90af47bf5d4f8"
format_stamp: "Formatted at 2026-09-26 05:50:57 on dist-test-slave-21p4"
server_key: "73968d1dbcd429d93fb7808911cbf747"
server_key_iv: "2fbdb6c1e6b933be7117adc0ab3c94fb"
server_key_version: "encryptionkey@0"
I20260926 05:50:57.659525 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "8948dae1c660429492a90af47bf5d4f8"
format_stamp: "Formatted at 2026-09-26 05:50:57 on dist-test-slave-21p4"
server_key: "73968d1dbcd429d93fb7808911cbf747"
server_key_iv: "2fbdb6c1e6b933be7117adc0ab3c94fb"
server_key_version: "encryptionkey@0"
I20260926 05:50:57.660969 761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:50:57.661770 776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:57.662005 761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:50:57.662122 761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "8948dae1c660429492a90af47bf5d4f8"
format_stamp: "Formatted at 2026-09-26 05:50:57 on dist-test-slave-21p4"
server_key: "73968d1dbcd429d93fb7808911cbf747"
server_key_iv: "2fbdb6c1e6b933be7117adc0ab3c94fb"
server_key_version: "encryptionkey@0"
I20260926 05:50:57.662202 761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:57.670990 761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:57.673035 761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:57.673199 761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:57.677286 761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33917
I20260926 05:50:57.677337 828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33917 every 8 connection(s)
I20260926 05:50:57.677719 761 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:57.678704 829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:57.679384 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 761
I20260926 05:50:57.679502 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:57.679705 12252 external_mini_cluster.cc:1428] Setting key 59bca73796fe03f3159daaa33be1dd6d
I20260926 05:50:57.681831 829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8: Bootstrap starting.
I20260926 05:50:57.682439 829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:57.682808 829 log.cc:824] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:57.683527 829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8: No bootstrap required, opened a new log
I20260926 05:50:57.685699 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:57.680232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:45377 (local address 127.0.0.1:33917)
0926 05:50:57.680415 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:57.680423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:57.680447 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:50:57.680897 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:57.680905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:57.680955 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:57.681065 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:57.681433 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:57.682429 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:57.683534 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:57.684118 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:57.684234 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:57.684973 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:57.684977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:57.684978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:57.685184 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:57.685223 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:57.685237 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:57.685339 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:57.685444 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 05:50:57.685474 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260926 05:50:57.686228 829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948dae1c660429492a90af47bf5d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33917 } }
I20260926 05:50:57.686411 829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:57.686460 829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8948dae1c660429492a90af47bf5d4f8, State: Initialized, Role: FOLLOWER
I20260926 05:50:57.686604 829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948dae1c660429492a90af47bf5d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33917 } }
I20260926 05:50:57.686705 829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:57.686753 829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:57.686808 829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:57.686831 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:57.687399 829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948dae1c660429492a90af47bf5d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33917 } }
I20260926 05:50:57.687518 829 leader_election.cc:302] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8948dae1c660429492a90af47bf5d4f8; no voters:
I20260926 05:50:57.687711 829 leader_election.cc:288] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:57.687762 834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:57.687958 834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [term 1 LEADER]: Becoming Leader. State: Replica: 8948dae1c660429492a90af47bf5d4f8, State: Running, Role: LEADER
I20260926 05:50:57.687992 829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:57.688082 834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948dae1c660429492a90af47bf5d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33917 } }
I20260926 05:50:57.688509 834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8948dae1c660429492a90af47bf5d4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948dae1c660429492a90af47bf5d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33917 } } }
I20260926 05:50:57.688655 834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:57.688843 834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8948dae1c660429492a90af47bf5d4f8. Latest consensus state: current_term: 1 leader_uuid: "8948dae1c660429492a90af47bf5d4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948dae1c660429492a90af47bf5d4f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33917 } } }
I20260926 05:50:57.688932 834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:57.689306 848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:57.689805 848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:57.691342 848 catalog_manager.cc:1347] Generated new cluster ID: 78cc466def5043158b30749105aad122
I20260926 05:50:57.691416 848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:57.691776 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:57.688228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:51941 (local address 127.0.0.1:33917)
0926 05:50:57.688266 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:57.688268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:57.688448 (+ 180us) server_negotiation.cc:407] Connection header received
0926 05:50:57.688573 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:57.688574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:57.688585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:57.688619 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:57.688838 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:57.689606 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:57.690412 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:50:57.690722 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:50:57.690788 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:50:57.691499 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:57.691500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:57.691500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:57.691505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:57.691520 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:57.691524 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:57.691557 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:57.691699 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 05:50:57.691710 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:50:57.711481 848 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:57.712074 848 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:57.720866 848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8948dae1c660429492a90af47bf5d4f8: Generated new TSK 0
I20260926 05:50:57.721066 848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:57.731071 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 761
W20260926 05:50:57.735728 746 connection.cc:535] client connection to 127.0.0.1:33917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33917): Connection reset by peer (error 104)
2026-09-26T05:50:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:57 dist-test-slave-21p4 krb5kdc[860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:57 dist-test-slave-21p4 krb5kdc[860](info): set up 2 sockets
Sep 26 05:50:57 dist-test-slave-21p4 krb5kdc[860](info): commencing operation
krb5kdc: starting...
W20260926 05:50:58.766985 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:50:58 dist-test-slave-21p4 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:50:58.893654 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:50:58.976738 873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:50:58.976960 873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:50:58.976985 873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:50:58.978169 873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:50:58.978214 873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:50:58.978227 873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:50:58.978240 873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:50:58.978252 873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:50:58.979843 873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:50:58.980110 873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:50:58.980351 873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:50:58.985483 878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:58.985464 881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:50:58.985541 879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:50:58.985702 873 server_base.cc:1034] running on GCE node
I20260926 05:50:58.985999 873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:50:58.986276 873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:50:58.987459 873 hybrid_clock.cc:648] HybridClock initialized: now 1790401858987435 us; error 41 us; skew 500 ppm
Sep 26 05:50:58 dist-test-slave-21p4 krb5kdc[860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:50:58.990198 873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:50:58.990796 873 webserver.cc:460] Webserver started at http://127.0.0.1:46487/ using document root <none> and password file <none>
I20260926 05:50:58.991062 873 fs_manager.cc:359] Metadata directory not provided
I20260926 05:50:58.991139 873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:50:58.991305 873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:50:58.992527 873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "6f9a30a0d9f04550be1d73b91e6e572a"
format_stamp: "Formatted at 2026-09-26 05:50:58 on dist-test-slave-21p4"
server_key: "82bad7f3e23892b6c22c0cbc0182b297"
server_key_iv: "45834373bae93cb3ffceb887264eca60"
server_key_version: "encryptionkey@0"
I20260926 05:50:58.992954 873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "6f9a30a0d9f04550be1d73b91e6e572a"
format_stamp: "Formatted at 2026-09-26 05:50:58 on dist-test-slave-21p4"
server_key: "82bad7f3e23892b6c22c0cbc0182b297"
server_key_iv: "45834373bae93cb3ffceb887264eca60"
server_key_version: "encryptionkey@0"
I20260926 05:50:58.994524 873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:50:58.995415 888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:50:58.995610 873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:50:58.995721 873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "6f9a30a0d9f04550be1d73b91e6e572a"
format_stamp: "Formatted at 2026-09-26 05:50:58 on dist-test-slave-21p4"
server_key: "82bad7f3e23892b6c22c0cbc0182b297"
server_key_iv: "45834373bae93cb3ffceb887264eca60"
server_key_version: "encryptionkey@0"
I20260926 05:50:58.995806 873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:50:58.999447 873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:50:59.001533 873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:50:59.001710 873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:50:59.006021 873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44643
I20260926 05:50:59.006057 940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44643 every 8 connection(s)
I20260926 05:50:59.006454 873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:50:59.007526 941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:50:59.008723 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 873
I20260926 05:50:59.008880 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:50:59.009101 12252 external_mini_cluster.cc:1428] Setting key a890fdd9c812b89ce80626962ba898bd
I20260926 05:50:59.010452 941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a: Bootstrap starting.
I20260926 05:50:59.011242 941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a: Neither blocks nor log segments found. Creating new log.
I20260926 05:50:59.011600 941 log.cc:824] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:50:59.011772 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:59.009685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:37913 (local address 127.0.0.1:44643)
0926 05:50:59.009869 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:59.009883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:59.009908 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:50:59.010321 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:59.010328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:59.010353 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:59.010501 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:59.011217 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:59.011221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:59.011221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:59.011224 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:59.011260 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:59.011270 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:59.011386 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:59.011504 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 05:50:59.011565 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260926 05:50:59.012331 941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a: No bootstrap required, opened a new log
I20260926 05:50:59.012781 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:50:59.014514 941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260926 05:50:59.014752 941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:50:59.014806 941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9a30a0d9f04550be1d73b91e6e572a, State: Initialized, Role: FOLLOWER
I20260926 05:50:59.014936 941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260926 05:50:59.015038 941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:50:59.015075 941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:50:59.015118 941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:50:59.015452 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:50:59.014179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:39601 (local address 127.0.0.1:44643)
0926 05:50:59.014214 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:50:59.014216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:50:59.014308 (+ 92us) server_negotiation.cc:407] Connection header received
0926 05:50:59.014372 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:50:59.014373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:50:59.014377 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:50:59.014424 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:50:59.015142 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:50:59.015143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:50:59.015144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:50:59.015148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:50:59.015203 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:50:59.015209 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:50:59.015286 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:50:59.015354 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 05:50:59.015368 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:50:59.015887 941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260926 05:50:59.015980 941 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9a30a0d9f04550be1d73b91e6e572a; no voters:
I20260926 05:50:59.016156 941 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:50:59.016211 951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:50:59.016374 951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [term 1 LEADER]: Becoming Leader. State: Replica: 6f9a30a0d9f04550be1d73b91e6e572a, State: Running, Role: LEADER
I20260926 05:50:59.016443 941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:50:59.016485 951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260926 05:50:59.016820 951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9a30a0d9f04550be1d73b91e6e572a. Latest consensus state: current_term: 1 leader_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260926 05:50:59.016837 953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9a30a0d9f04550be1d73b91e6e572a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260926 05:50:59.016939 951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:59.016947 953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a [sys.catalog]: This master's current role is: LEADER
I20260926 05:50:59.017197 957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:50:59.017822 957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:50:59.019562 957 catalog_manager.cc:1347] Generated new cluster ID: 1cd5946ec88f4c968b87534b7d44dd33
I20260926 05:50:59.019620 957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:50:59.042462 957 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:50:59.042714 957 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:50:59.044962 957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f9a30a0d9f04550be1d73b91e6e572a: Generated new TSK 0
I20260926 05:50:59.045162 957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:50:59.054330 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 873
2026-09-26T05:50:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:50:59 dist-test-slave-21p4 krb5kdc[972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:50:59 dist-test-slave-21p4 krb5kdc[972](info): set up 2 sockets
Sep 26 05:50:59 dist-test-slave-21p4 krb5kdc[972](info): commencing operation
krb5kdc: starting...
W20260926 05:51:00.098857 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:00 dist-test-slave-21p4 krb5kdc[972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:51:00.222447 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:00.306308 985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:00.306578 985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:00.306635 985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:00.307961 985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:00.308045 985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:00.308072 985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:00.308099 985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:00.308131 985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:00.309701 985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:00.310004 985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:00.310251 985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:00.315305 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:00.315316 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:00.315307 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:00.315342 985 server_base.cc:1034] running on GCE node
I20260926 05:51:00.315834 985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:00.316147 985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:00.317353 985 hybrid_clock.cc:648] HybridClock initialized: now 1790401860317349 us; error 64 us; skew 500 ppm
Sep 26 05:51:00 dist-test-slave-21p4 krb5kdc[972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:00.320091 985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:51:00.320577 985 webserver.cc:460] Webserver started at http://127.0.0.1:41841/ using document root <none> and password file <none>
I20260926 05:51:00.320837 985 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:00.320909 985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:00.321058 985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:00.322155 985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "e06da3d2ad414c30b80e6200970e6b7a"
format_stamp: "Formatted at 2026-09-26 05:51:00 on dist-test-slave-21p4"
server_key: "f5c781b47190021f5b4f166b7cde4f0d"
server_key_iv: "0fdb1424faca6460a04545764178979d"
server_key_version: "encryptionkey@0"
I20260926 05:51:00.322538 985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "e06da3d2ad414c30b80e6200970e6b7a"
format_stamp: "Formatted at 2026-09-26 05:51:00 on dist-test-slave-21p4"
server_key: "f5c781b47190021f5b4f166b7cde4f0d"
server_key_iv: "0fdb1424faca6460a04545764178979d"
server_key_version: "encryptionkey@0"
I20260926 05:51:00.323940 985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:00.324713 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:00.324882 985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:00.324977 985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "e06da3d2ad414c30b80e6200970e6b7a"
format_stamp: "Formatted at 2026-09-26 05:51:00 on dist-test-slave-21p4"
server_key: "f5c781b47190021f5b4f166b7cde4f0d"
server_key_iv: "0fdb1424faca6460a04545764178979d"
server_key_version: "encryptionkey@0"
I20260926 05:51:00.325074 985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:00.335420 985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:00.337764 985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:00.337884 985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:00.341856 985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35483
I20260926 05:51:00.341934 1052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35483 every 8 connection(s)
I20260926 05:51:00.342288 985 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:00.343271 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:00.345620 1053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a: Bootstrap starting.
I20260926 05:51:00.346230 1053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:00.346524 1053 log.cc:824] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:00.347259 1053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a: No bootstrap required, opened a new log
I20260926 05:51:00.347550 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 985
I20260926 05:51:00.347679 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:00.347922 12252 external_mini_cluster.cc:1428] Setting key dfedab9e5bba283571653c4156f46527
I20260926 05:51:00.349745 1053 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06da3d2ad414c30b80e6200970e6b7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260926 05:51:00.349901 1053 raft_consensus.cc:374] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:00.349953 1053 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06da3d2ad414c30b80e6200970e6b7a, State: Initialized, Role: FOLLOWER
I20260926 05:51:00.350091 1053 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06da3d2ad414c30b80e6200970e6b7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260926 05:51:00.350193 1053 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:00.350226 1053 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:00.350265 1053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:00.351085 1053 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06da3d2ad414c30b80e6200970e6b7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260926 05:51:00.351194 1053 leader_election.cc:302] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06da3d2ad414c30b80e6200970e6b7a; no voters:
I20260926 05:51:00.351349 1053 leader_election.cc:288] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:00.351411 1058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:00.351584 1058 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [term 1 LEADER]: Becoming Leader. State: Replica: e06da3d2ad414c30b80e6200970e6b7a, State: Running, Role: LEADER
I20260926 05:51:00.351588 1053 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:00.351708 1058 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06da3d2ad414c30b80e6200970e6b7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260926 05:51:00.352049 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06da3d2ad414c30b80e6200970e6b7a. Latest consensus state: current_term: 1 leader_uuid: "e06da3d2ad414c30b80e6200970e6b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06da3d2ad414c30b80e6200970e6b7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } } }
I20260926 05:51:00.352274 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:00.352202 1059 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e06da3d2ad414c30b80e6200970e6b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06da3d2ad414c30b80e6200970e6b7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } } }
I20260926 05:51:00.352759 1059 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a [sys.catalog]: This master's current role is: LEADER
W20260926 05:51:00.353500 1074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:00.353564 1074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:00.353657 1067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:00.353856 1067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:00.355587 1067 catalog_manager.cc:1347] Generated new cluster ID: 6e3ff62c547146a0936623c8714b73dc
I20260926 05:51:00.355654 1067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:00 dist-test-slave-21p4 krb5kdc[972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:51:00.360041 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:00.348519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:43123 (local address 127.0.0.1:35483)
0926 05:51:00.348729 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:00.348740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:00.348774 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:51:00.349319 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:00.349329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:00.349396 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:00.349565 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:00.350227 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:00.352037 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:00.353319 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:00.353913 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:00.354014 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:00.356795 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:00.356800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:00.356800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:00.356812 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:00.358499 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:00.358936 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:00.358939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:00.358939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:00.359028 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:00.359262 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:00.359263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:00.359264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:00.359501 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:00.359691 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:00.359783 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 05:51:00.359881 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260926 05:51:00.361052 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:00.368742 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:00.362394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:46589 (local address 127.0.0.1:35483)
0926 05:51:00.362439 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:00.362441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:00.362804 (+ 363us) server_negotiation.cc:407] Connection header received
0926 05:51:00.362885 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:00.362886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:00.362898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:00.362951 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:00.363447 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:00.363949 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:00.364645 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:00.364874 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:00.364952 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:00.366130 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:00.366132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:00.366132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:00.366140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:00.367548 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:00.367929 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:00.367930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:00.367931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:00.367984 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:00.368212 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:00.368213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:00.368213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:00.368334 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:00.368445 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:00.368580 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 05:51:00.368640 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:51:00.401527 1067 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:00.402249 1067 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:00.411755 1067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e06da3d2ad414c30b80e6200970e6b7a: Generated new TSK 0
I20260926 05:51:00.411962 1067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:00.415462 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 985
W20260926 05:51:00.420032 970 connection.cc:535] client connection to 127.0.0.1:35483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35483): Connection reset by peer (error 104)
2026-09-26T05:51:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:00 dist-test-slave-21p4 krb5kdc[1084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:00 dist-test-slave-21p4 krb5kdc[1084](info): set up 2 sockets
Sep 26 05:51:00 dist-test-slave-21p4 krb5kdc[1084](info): commencing operation
krb5kdc: starting...
W20260926 05:51:01.454900 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:01 dist-test-slave-21p4 krb5kdc[1084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:51:01.578397 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:01.663666 1097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:01.663902 1097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:01.663926 1097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:01.665206 1097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:01.665261 1097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:01.665275 1097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:01.665288 1097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:01.665300 1097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:01.666908 1097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:01.667182 1097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:01.667459 1097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:01.672627 1105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:01.672727 1097 server_base.cc:1034] running on GCE node
W20260926 05:51:01.672616 1102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:01.672621 1103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:01.673185 1097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:01.673473 1097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:01.674685 1097 hybrid_clock.cc:648] HybridClock initialized: now 1790401861674666 us; error 49 us; skew 500 ppm
Sep 26 05:51:01 dist-test-slave-21p4 krb5kdc[1084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:01.681841 1097 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:51:01.682510 1097 webserver.cc:460] Webserver started at http://127.0.0.1:35791/ using document root <none> and password file <none>
I20260926 05:51:01.682838 1097 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:01.682909 1097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:01.683064 1097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:01.684254 1097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "58a42a25a2d9428ba52df6b160a46642"
format_stamp: "Formatted at 2026-09-26 05:51:01 on dist-test-slave-21p4"
server_key: "6a13f50bc21e733dd5f2a7208d0d8f92"
server_key_iv: "e55229ea6c4b28d01b7d1494005db34b"
server_key_version: "encryptionkey@0"
I20260926 05:51:01.684656 1097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "58a42a25a2d9428ba52df6b160a46642"
format_stamp: "Formatted at 2026-09-26 05:51:01 on dist-test-slave-21p4"
server_key: "6a13f50bc21e733dd5f2a7208d0d8f92"
server_key_iv: "e55229ea6c4b28d01b7d1494005db34b"
server_key_version: "encryptionkey@0"
I20260926 05:51:01.686149 1097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:01.687112 1112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:01.687336 1097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:01.687438 1097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "58a42a25a2d9428ba52df6b160a46642"
format_stamp: "Formatted at 2026-09-26 05:51:01 on dist-test-slave-21p4"
server_key: "6a13f50bc21e733dd5f2a7208d0d8f92"
server_key_iv: "e55229ea6c4b28d01b7d1494005db34b"
server_key_version: "encryptionkey@0"
I20260926 05:51:01.687520 1097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:01.720549 1097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:01.723016 1097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:01.723182 1097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:01.727259 1097 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36087
I20260926 05:51:01.727308 1164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36087 every 8 connection(s)
I20260926 05:51:01.727707 1097 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:01.728612 1165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:01.731015 1165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642: Bootstrap starting.
I20260926 05:51:01.731617 1165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:01.731943 1165 log.cc:824] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:01.732754 1165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642: No bootstrap required, opened a new log
I20260926 05:51:01.734078 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1097
I20260926 05:51:01.734220 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:01.734424 12252 external_mini_cluster.cc:1428] Setting key 4039df21e8345917ffd88d0aa727a5b8
I20260926 05:51:01.734923 1165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a42a25a2d9428ba52df6b160a46642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260926 05:51:01.735142 1165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:01.735225 1165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a42a25a2d9428ba52df6b160a46642, State: Initialized, Role: FOLLOWER
I20260926 05:51:01.735396 1165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a42a25a2d9428ba52df6b160a46642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260926 05:51:01.735476 1165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:01.735498 1165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:01.735569 1165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:01.736538 1165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a42a25a2d9428ba52df6b160a46642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260926 05:51:01.736630 1165 leader_election.cc:302] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a42a25a2d9428ba52df6b160a46642; no voters:
I20260926 05:51:01.736832 1165 leader_election.cc:288] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:01.736855 1170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:01.736976 1170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [term 1 LEADER]: Becoming Leader. State: Replica: 58a42a25a2d9428ba52df6b160a46642, State: Running, Role: LEADER
I20260926 05:51:01.737120 1165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:01.737121 1170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a42a25a2d9428ba52df6b160a46642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260926 05:51:01.738379 1171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a42a25a2d9428ba52df6b160a46642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a42a25a2d9428ba52df6b160a46642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } } }
I20260926 05:51:01.738485 1171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [sys.catalog]: This master's current role is: LEADER
W20260926 05:51:01.739171 1185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 05:51:01.739202 1172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a42a25a2d9428ba52df6b160a46642. Latest consensus state: current_term: 1 leader_uuid: "58a42a25a2d9428ba52df6b160a46642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a42a25a2d9428ba52df6b160a46642" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } } }
W20260926 05:51:01.739619 1185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:01.739674 1172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:01.739720 1186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:01.739904 1186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:01.740478 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:01.735090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:37295 (local address 127.0.0.1:36087)
0926 05:51:01.735310 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:01.735321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:01.735357 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:51:01.735507 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:01.735515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:01.735579 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:01.735720 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:01.735956 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:01.737025 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:01.738169 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:01.738860 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:01.738986 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:01.739784 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:01.739788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:01.739789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:01.740018 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:01.740057 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:01.740078 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:01.740175 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:01.740260 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 05:51:01.740290 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260926 05:51:01.741422 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:01.741496 1186 catalog_manager.cc:1347] Generated new cluster ID: cebb1ee4c17342a8a90c5edfebe3de6c
I20260926 05:51:01.741549 1186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:01.754518 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:01.750713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:52273 (local address 127.0.0.1:36087)
0926 05:51:01.750794 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:01.750797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:01.750812 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:51:01.750845 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:01.750847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:01.750860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:01.750954 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:01.751191 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:01.751992 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:01.753029 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:01.753374 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:01.753448 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:01.754219 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:01.754221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:01.754221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:01.754227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:01.754245 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:01.754250 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:01.754315 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:01.754404 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 05:51:01.754419 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260926 05:51:01.759804 1186 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:01.761137 1186 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:01.764274 1186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a42a25a2d9428ba52df6b160a46642: Generated new TSK 0
I20260926 05:51:01.764473 1186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:01.767246 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1097
W20260926 05:51:01.771821 1082 connection.cc:535] client connection to 127.0.0.1:36087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36087): Connection reset by peer (error 104)
2026-09-26T05:51:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:01 dist-test-slave-21p4 krb5kdc[1196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:01 dist-test-slave-21p4 krb5kdc[1196](info): set up 2 sockets
Sep 26 05:51:01 dist-test-slave-21p4 krb5kdc[1196](info): commencing operation
krb5kdc: starting...
W20260926 05:51:02.802912 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:02 dist-test-slave-21p4 krb5kdc[1196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:51:02.929761 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:03.011232 1209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:03.011449 1209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:03.011472 1209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:03.012653 1209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:03.012703 1209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:03.012717 1209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:03.012730 1209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:03.012742 1209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:03.014295 1209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:03.014569 1209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:03.014846 1209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:03.019806 1215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:03.019909 1217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:03.019806 1214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:03.020077 1209 server_base.cc:1034] running on GCE node
I20260926 05:51:03.020375 1209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:03.020715 1209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:03.021896 1209 hybrid_clock.cc:648] HybridClock initialized: now 1790401863021883 us; error 43 us; skew 500 ppm
Sep 26 05:51:03 dist-test-slave-21p4 krb5kdc[1196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:03.024510 1209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:51:03.025023 1209 webserver.cc:460] Webserver started at http://127.0.0.1:38357/ using document root <none> and password file <none>
I20260926 05:51:03.025321 1209 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:03.025393 1209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:03.025552 1209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:03.026708 1209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "e2c1946948374e67b9c4ec50496de532"
format_stamp: "Formatted at 2026-09-26 05:51:03 on dist-test-slave-21p4"
server_key: "bc2d36606f7d5e59267f7af48a1ca4b8"
server_key_iv: "80bb589af449a8d4343c18ca9a133f90"
server_key_version: "encryptionkey@0"
I20260926 05:51:03.027086 1209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "e2c1946948374e67b9c4ec50496de532"
format_stamp: "Formatted at 2026-09-26 05:51:03 on dist-test-slave-21p4"
server_key: "bc2d36606f7d5e59267f7af48a1ca4b8"
server_key_iv: "80bb589af449a8d4343c18ca9a133f90"
server_key_version: "encryptionkey@0"
I20260926 05:51:03.028442 1209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:03.029249 1224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:03.029428 1209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:03.029522 1209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "e2c1946948374e67b9c4ec50496de532"
format_stamp: "Formatted at 2026-09-26 05:51:03 on dist-test-slave-21p4"
server_key: "bc2d36606f7d5e59267f7af48a1ca4b8"
server_key_iv: "80bb589af449a8d4343c18ca9a133f90"
server_key_version: "encryptionkey@0"
I20260926 05:51:03.029623 1209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:03.033103 1209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:03.035105 1209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:03.035264 1209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:03.039217 1209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32885
I20260926 05:51:03.039266 1276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32885 every 8 connection(s)
I20260926 05:51:03.039619 1209 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:03.040546 1277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:03.043026 1277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532: Bootstrap starting.
I20260926 05:51:03.043624 1277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:03.043915 1277 log.cc:824] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:03.044615 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1209
I20260926 05:51:03.044735 1277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532: No bootstrap required, opened a new log
I20260926 05:51:03.044742 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:03.044966 12252 external_mini_cluster.cc:1428] Setting key 96071c4a455774730c5550dea0368e92
I20260926 05:51:03.046979 1277 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c1946948374e67b9c4ec50496de532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260926 05:51:03.047170 1277 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:03.047209 1277 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c1946948374e67b9c4ec50496de532, State: Initialized, Role: FOLLOWER
I20260926 05:51:03.047405 1277 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c1946948374e67b9c4ec50496de532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260926 05:51:03.047535 1277 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:03.047595 1277 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:03.047672 1277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:03.048368 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:03.045540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:45739 (local address 127.0.0.1:32885)
0926 05:51:03.045755 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:03.045764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:03.045800 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:51:03.046924 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:03.046934 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:03.046974 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:03.047138 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:03.047875 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:03.047879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:03.047879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:03.047884 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:03.047929 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:03.047944 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:03.048018 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:03.048080 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 05:51:03.048122 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260926 05:51:03.048923 1277 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c1946948374e67b9c4ec50496de532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260926 05:51:03.049043 1277 leader_election.cc:302] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c1946948374e67b9c4ec50496de532; no voters:
I20260926 05:51:03.049207 1277 leader_election.cc:288] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:03.049259 1282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:03.049401 1282 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [term 1 LEADER]: Becoming Leader. State: Replica: e2c1946948374e67b9c4ec50496de532, State: Running, Role: LEADER
I20260926 05:51:03.049459 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:03.049467 1277 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:03.049503 1282 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c1946948374e67b9c4ec50496de532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } }
I20260926 05:51:03.049794 1282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c1946948374e67b9c4ec50496de532. Latest consensus state: current_term: 1 leader_uuid: "e2c1946948374e67b9c4ec50496de532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c1946948374e67b9c4ec50496de532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } } }
I20260926 05:51:03.049794 1284 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2c1946948374e67b9c4ec50496de532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c1946948374e67b9c4ec50496de532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32885 } } }
I20260926 05:51:03.049912 1282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:03.049918 1284 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:03.050091 1288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:03.050665 1288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:03.052597 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:03.051160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:60089 (local address 127.0.0.1:32885)
0926 05:51:03.051198 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:03.051200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:03.051519 (+ 319us) server_negotiation.cc:407] Connection header received
0926 05:51:03.051614 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:03.051615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:03.051631 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:03.051662 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:03.052329 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:03.052330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:03.052331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:03.052335 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:03.052352 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:03.052356 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:03.052421 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:03.052517 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:51:03.052535 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:51:03.052747 1288 catalog_manager.cc:1347] Generated new cluster ID: 7180d494c4a04976abd0b1d1d5dcc993
I20260926 05:51:03.052789 1288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:03.063308 1288 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:03.063521 1288 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:03.067531 1288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c1946948374e67b9c4ec50496de532: Generated new TSK 0
I20260926 05:51:03.067739 1288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:03.073167 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1209
2026-09-26T05:51:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:03 dist-test-slave-21p4 krb5kdc[1308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:03 dist-test-slave-21p4 krb5kdc[1308](info): set up 2 sockets
Sep 26 05:51:03 dist-test-slave-21p4 krb5kdc[1308](info): commencing operation
krb5kdc: starting...
W20260926 05:51:04.114668 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:04 dist-test-slave-21p4 krb5kdc[1308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:04.237879 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:38009
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:38009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:04.319084 1321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:04.319295 1321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:04.319319 1321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:04.320411 1321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:04.320446 1321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:04.320461 1321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:04.320473 1321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:04.320485 1321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:04.321918 1321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:38009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:38009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:04.322110 1321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:04.322281 1321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:04.327184 1327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:04.327190 1329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:04.327195 1326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:04.327481 1321 server_base.cc:1034] running on GCE node
I20260926 05:51:04.327723 1321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:04.327967 1321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:04.329135 1321 hybrid_clock.cc:648] HybridClock initialized: now 1790401864329110 us; error 35 us; skew 500 ppm
Sep 26 05:51:04 dist-test-slave-21p4 krb5kdc[1308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401864, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:04.331947 1321 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:04.332427 1321 webserver.cc:460] Webserver started at http://127.11.247.62:41023/ using document root <none> and password file <none>
I20260926 05:51:04.332661 1321 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:04.332728 1321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:04.332871 1321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:04.334007 1321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "0e0c6a47c80a46058e9f9446220ee227"
format_stamp: "Formatted at 2026-09-26 05:51:04 on dist-test-slave-21p4"
server_key: "bca14364a425119850bb513d0736e45a"
server_key_iv: "609b42ece8200e9bb13b5abe4b7b9821"
server_key_version: "encryptionkey@0"
I20260926 05:51:04.334376 1321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "0e0c6a47c80a46058e9f9446220ee227"
format_stamp: "Formatted at 2026-09-26 05:51:04 on dist-test-slave-21p4"
server_key: "bca14364a425119850bb513d0736e45a"
server_key_iv: "609b42ece8200e9bb13b5abe4b7b9821"
server_key_version: "encryptionkey@0"
I20260926 05:51:04.335768 1321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:51:04.336594 1336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:04.336836 1321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:04.336954 1321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "0e0c6a47c80a46058e9f9446220ee227"
format_stamp: "Formatted at 2026-09-26 05:51:04 on dist-test-slave-21p4"
server_key: "bca14364a425119850bb513d0736e45a"
server_key_iv: "609b42ece8200e9bb13b5abe4b7b9821"
server_key_version: "encryptionkey@0"
I20260926 05:51:04.337021 1321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:04.349079 1321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:04.351313 1321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:04.351434 1321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:04.355937 1321 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:38009
I20260926 05:51:04.355985 1388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:38009 every 8 connection(s)
I20260926 05:51:04.356348 1321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:04.357262 1389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:04.360080 1389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227: Bootstrap starting.
I20260926 05:51:04.360674 1389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:04.360955 1389 log.cc:824] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:04.361799 1389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227: No bootstrap required, opened a new log
I20260926 05:51:04.363166 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1321
I20260926 05:51:04.363306 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:04.363518 12252 external_mini_cluster.cc:1428] Setting key 968b694e8e0f3bb27a917b172d1cce70
I20260926 05:51:04.364001 1389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0c6a47c80a46058e9f9446220ee227" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 38009 } }
I20260926 05:51:04.364228 1389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:04.364291 1389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0c6a47c80a46058e9f9446220ee227, State: Initialized, Role: FOLLOWER
I20260926 05:51:04.364449 1389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0c6a47c80a46058e9f9446220ee227" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 38009 } }
I20260926 05:51:04.364542 1389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:04.364578 1389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:04.364631 1389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:04.365813 1389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0c6a47c80a46058e9f9446220ee227" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 38009 } }
I20260926 05:51:04.365934 1389 leader_election.cc:302] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e0c6a47c80a46058e9f9446220ee227; no voters:
I20260926 05:51:04.366114 1389 leader_election.cc:288] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:04.366164 1394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:04.366339 1394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [term 1 LEADER]: Becoming Leader. State: Replica: 0e0c6a47c80a46058e9f9446220ee227, State: Running, Role: LEADER
I20260926 05:51:04.366510 1394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0c6a47c80a46058e9f9446220ee227" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 38009 } }
I20260926 05:51:04.366381 1389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:04.366922 1396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e0c6a47c80a46058e9f9446220ee227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0c6a47c80a46058e9f9446220ee227" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 38009 } } }
I20260926 05:51:04.366950 1395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e0c6a47c80a46058e9f9446220ee227. Latest consensus state: current_term: 1 leader_uuid: "0e0c6a47c80a46058e9f9446220ee227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0c6a47c80a46058e9f9446220ee227" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 38009 } } }
I20260926 05:51:04.367070 1396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:04.367169 1395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:04.367421 1398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:04.368096 1398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:04.369984 1398 catalog_manager.cc:1347] Generated new cluster ID: 93b12191740e46bcb7b552a727dc78be
I20260926 05:51:04.370049 1398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:04 dist-test-slave-21p4 krb5kdc[1308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:04.374841 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:04.364135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44973 (local address 127.11.247.62:38009)
0926 05:51:04.364410 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:04.364420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:04.364453 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:51:04.364572 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:04.364580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:04.364639 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:04.364787 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:04.365410 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:04.366889 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:04.368077 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:04.368782 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:04.368905 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:04.371528 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:04.371532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:04.371532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:04.371545 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:04.373227 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:04.373630 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:04.373633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:04.373633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:04.373742 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:04.373972 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:04.373974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:04.373974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:04.374216 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:04.374414 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:04.374501 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:51:04.374639 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260926 05:51:04.376120 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:04.384514 1393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:04.377843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46033 (local address 127.11.247.62:38009)
0926 05:51:04.377879 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:04.377881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:04.378156 (+ 275us) server_negotiation.cc:407] Connection header received
0926 05:51:04.378254 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:04.378255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:04.378266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:04.378316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:04.378969 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:04.379495 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:04.380199 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:04.380437 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:04.380523 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:04.381735 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:04.381736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:04.381737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:04.381744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:04.383309 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:04.383668 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:04.383669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:04.383669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:04.383729 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:04.383961 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:04.383963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:04.383963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:04.384081 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:04.384207 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:04.384328 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 05:51:04.384393 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:51:04.396934 1398 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:04.397559 1398 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:04.403739 1398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e0c6a47c80a46058e9f9446220ee227: Generated new TSK 0
I20260926 05:51:04.403986 1398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:04.413160 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1321
W20260926 05:51:04.417970 1306 connection.cc:535] client connection to 127.11.247.62:38009 recv error: Network error: failed to read from TLS socket (remote: 127.11.247.62:38009): Connection reset by peer (error 104)
2026-09-26T05:51:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:04 dist-test-slave-21p4 krb5kdc[1420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:04 dist-test-slave-21p4 krb5kdc[1420](info): set up 2 sockets
Sep 26 05:51:04 dist-test-slave-21p4 krb5kdc[1420](info): commencing operation
krb5kdc: starting...
W20260926 05:51:05.446832 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:05 dist-test-slave-21p4 krb5kdc[1420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:05.574723 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:35387
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:05.659520 1433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:05.659772 1433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:05.659811 1433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:05.661026 1433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:05.661108 1433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:05.661134 1433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:05.661166 1433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:05.661190 1433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:05.662792 1433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:35387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:35387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:05.663087 1433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:05.663328 1433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:05.668488 1438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:05.668538 1439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:05.668534 1433 server_base.cc:1034] running on GCE node
W20260926 05:51:05.668502 1441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:05.669077 1433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:05.669430 1433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:05.670646 1433 hybrid_clock.cc:648] HybridClock initialized: now 1790401865670598 us; error 79 us; skew 500 ppm
Sep 26 05:51:05 dist-test-slave-21p4 krb5kdc[1420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401865, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:05.673187 1433 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:05.673619 1433 webserver.cc:460] Webserver started at http://127.11.247.62:45737/ using document root <none> and password file <none>
I20260926 05:51:05.673874 1433 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:05.673943 1433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:05.674089 1433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:05.675247 1433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "dec84947199c4207a1a5b0439c9ab3c9"
format_stamp: "Formatted at 2026-09-26 05:51:05 on dist-test-slave-21p4"
server_key: "df3d90a86e6f446da4cd59e50dc768f5"
server_key_iv: "64e687d3e2d3ca4611f481d701e2b11e"
server_key_version: "encryptionkey@0"
I20260926 05:51:05.675602 1433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "dec84947199c4207a1a5b0439c9ab3c9"
format_stamp: "Formatted at 2026-09-26 05:51:05 on dist-test-slave-21p4"
server_key: "df3d90a86e6f446da4cd59e50dc768f5"
server_key_iv: "64e687d3e2d3ca4611f481d701e2b11e"
server_key_version: "encryptionkey@0"
I20260926 05:51:05.676927 1433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:05.677611 1448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:05.677774 1433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:05.677870 1433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "dec84947199c4207a1a5b0439c9ab3c9"
format_stamp: "Formatted at 2026-09-26 05:51:05 on dist-test-slave-21p4"
server_key: "df3d90a86e6f446da4cd59e50dc768f5"
server_key_iv: "64e687d3e2d3ca4611f481d701e2b11e"
server_key_version: "encryptionkey@0"
I20260926 05:51:05.677955 1433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:05.702983 1433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:05.705158 1433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:05.705349 1433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:05.709358 1433 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:35387
I20260926 05:51:05.709403 1500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:35387 every 8 connection(s)
I20260926 05:51:05.709753 1433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:05.710021 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1433
I20260926 05:51:05.710157 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:05.710386 12252 external_mini_cluster.cc:1428] Setting key f517ba8244456e478ee773cf27ed42df
I20260926 05:51:05.710867 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:05.714186 1501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9: Bootstrap starting.
I20260926 05:51:05.714887 1501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:05.715175 1501 log.cc:824] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:05.715934 1501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9: No bootstrap required, opened a new log
I20260926 05:51:05.716333 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:05.710958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60821 (local address 127.11.247.62:35387)
0926 05:51:05.711154 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:05.711163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:05.711195 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:51:05.711732 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:05.711741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:05.711806 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:05.711970 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:05.712146 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:05.713523 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:05.714311 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:05.714799 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:05.714921 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:05.715744 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:05.715748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:05.715748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:05.715905 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:05.715942 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:05.715953 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:05.716052 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:05.716135 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:51:05.716163 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260926 05:51:05.717375 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:05.718314 1501 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec84947199c4207a1a5b0439c9ab3c9" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35387 } }
I20260926 05:51:05.718504 1501 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:05.718559 1501 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec84947199c4207a1a5b0439c9ab3c9, State: Initialized, Role: FOLLOWER
I20260926 05:51:05.718714 1501 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec84947199c4207a1a5b0439c9ab3c9" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35387 } }
I20260926 05:51:05.718794 1501 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:05.718847 1501 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:05.718899 1501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:05.719498 1501 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec84947199c4207a1a5b0439c9ab3c9" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35387 } }
I20260926 05:51:05.719620 1501 leader_election.cc:302] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec84947199c4207a1a5b0439c9ab3c9; no voters:
I20260926 05:51:05.719802 1501 leader_election.cc:288] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:05.719863 1511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:05.720058 1501 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:05.720081 1511 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [term 1 LEADER]: Becoming Leader. State: Replica: dec84947199c4207a1a5b0439c9ab3c9, State: Running, Role: LEADER
I20260926 05:51:05.720281 1511 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec84947199c4207a1a5b0439c9ab3c9" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35387 } }
I20260926 05:51:05.720643 1512 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec84947199c4207a1a5b0439c9ab3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec84947199c4207a1a5b0439c9ab3c9" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35387 } } }
I20260926 05:51:05.720714 1512 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:05.720940 1513 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec84947199c4207a1a5b0439c9ab3c9. Latest consensus state: current_term: 1 leader_uuid: "dec84947199c4207a1a5b0439c9ab3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec84947199c4207a1a5b0439c9ab3c9" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 35387 } } }
I20260926 05:51:05.721045 1513 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:05.721150 1520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:05.722095 1520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:05.722378 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:05.718740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56321 (local address 127.11.247.62:35387)
0926 05:51:05.718784 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:05.718786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:05.718953 (+ 167us) server_negotiation.cc:407] Connection header received
0926 05:51:05.719010 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:05.719012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:05.719022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:05.719066 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:05.719213 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:05.719782 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:05.720901 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:05.721222 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:05.721280 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:05.722098 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:05.722099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:05.722099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:05.722104 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:05.722118 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:05.722122 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:05.722156 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:05.722300 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:51:05.722314 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:51:05.723830 1520 catalog_manager.cc:1347] Generated new cluster ID: 5c8e31194cfa49cbad4beceda6cfeeed
I20260926 05:51:05.723888 1520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:05.747823 1520 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:05.748428 1520 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:05.761442 1520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec84947199c4207a1a5b0439c9ab3c9: Generated new TSK 0
I20260926 05:51:05.761668 1520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:05.763891 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1433
W20260926 05:51:05.768293 1418 connection.cc:535] client connection to 127.11.247.62:35387 recv error: Network error: failed to read from TLS socket (remote: 127.11.247.62:35387): Connection reset by peer (error 104)
2026-09-26T05:51:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:05 dist-test-slave-21p4 krb5kdc[1532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:05 dist-test-slave-21p4 krb5kdc[1532](info): set up 2 sockets
Sep 26 05:51:05 dist-test-slave-21p4 krb5kdc[1532](info): commencing operation
krb5kdc: starting...
W20260926 05:51:06.826799 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:06 dist-test-slave-21p4 krb5kdc[1532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:06.958901 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:44467
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:07.046945 1545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:07.047224 1545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:07.047271 1545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:07.048578 1545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:07.048666 1545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:07.048689 1545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:07.048719 1545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:07.048751 1545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:07.050377 1545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:44467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:44467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:07.050709 1545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:07.050964 1545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:07.056380 1550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:07.056377 1551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:07.056396 1545 server_base.cc:1034] running on GCE node
W20260926 05:51:07.056368 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:07.056880 1545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:07.057199 1545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:07.058377 1545 hybrid_clock.cc:648] HybridClock initialized: now 1790401867058346 us; error 44 us; skew 500 ppm
Sep 26 05:51:07 dist-test-slave-21p4 krb5kdc[1532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401867, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:07.067420 1545 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:07.067920 1545 webserver.cc:460] Webserver started at http://127.11.247.62:45449/ using document root <none> and password file <none>
I20260926 05:51:07.068184 1545 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:07.068257 1545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:07.068437 1545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:07.069636 1545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "650506d148f441a987cdb3a00e87ee79"
format_stamp: "Formatted at 2026-09-26 05:51:07 on dist-test-slave-21p4"
server_key: "7f2f0fcb1ff844feeb8143723df1f305"
server_key_iv: "00dbd7fc018fe6a671000081d82b784e"
server_key_version: "encryptionkey@0"
I20260926 05:51:07.070034 1545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "650506d148f441a987cdb3a00e87ee79"
format_stamp: "Formatted at 2026-09-26 05:51:07 on dist-test-slave-21p4"
server_key: "7f2f0fcb1ff844feeb8143723df1f305"
server_key_iv: "00dbd7fc018fe6a671000081d82b784e"
server_key_version: "encryptionkey@0"
I20260926 05:51:07.071569 1545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:51:07.072536 1560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:07.072782 1545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:07.072888 1545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "650506d148f441a987cdb3a00e87ee79"
format_stamp: "Formatted at 2026-09-26 05:51:07 on dist-test-slave-21p4"
server_key: "7f2f0fcb1ff844feeb8143723df1f305"
server_key_iv: "00dbd7fc018fe6a671000081d82b784e"
server_key_version: "encryptionkey@0"
I20260926 05:51:07.072996 1545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:07.076680 1545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:07.079068 1545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:07.079243 1545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:07.083520 1545 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:44467
I20260926 05:51:07.083554 1613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:44467 every 8 connection(s)
I20260926 05:51:07.083925 1545 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:07.084415 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1545
I20260926 05:51:07.084538 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:07.084741 12252 external_mini_cluster.cc:1428] Setting key 550525e135d26ed4c1ab695817dbd92f
I20260926 05:51:07.085182 1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:07.087338 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:07.085299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33799 (local address 127.11.247.62:44467)
0926 05:51:07.085493 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:07.085501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:07.085526 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:51:07.085967 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:07.085974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:07.085999 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:07.086120 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:07.086864 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:07.086868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:07.086868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:07.086872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:07.086911 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:07.086921 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:07.086977 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:07.087078 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:51:07.087134 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260926 05:51:07.088404 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:07.088991 1614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79: Bootstrap starting.
I20260926 05:51:07.089783 1614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:07.090152 1614 log.cc:824] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:07.090973 1614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79: No bootstrap required, opened a new log
I20260926 05:51:07.091367 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:07.090085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55199 (local address 127.11.247.62:44467)
0926 05:51:07.090151 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:07.090153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:07.090209 (+ 56us) server_negotiation.cc:407] Connection header received
0926 05:51:07.090317 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:07.090318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:07.090323 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:07.090372 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:07.091164 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:07.091165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:07.091165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:07.091171 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:07.091185 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:07.091190 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:07.091258 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:07.091295 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 05:51:07.091308 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:51:07.093055 1614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650506d148f441a987cdb3a00e87ee79" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44467 } }
I20260926 05:51:07.093216 1614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:07.093266 1614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650506d148f441a987cdb3a00e87ee79, State: Initialized, Role: FOLLOWER
I20260926 05:51:07.093387 1614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650506d148f441a987cdb3a00e87ee79" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44467 } }
I20260926 05:51:07.093498 1614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:07.093547 1614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:07.093600 1614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:07.094185 1614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650506d148f441a987cdb3a00e87ee79" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44467 } }
I20260926 05:51:07.094310 1614 leader_election.cc:302] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650506d148f441a987cdb3a00e87ee79; no voters:
I20260926 05:51:07.094496 1614 leader_election.cc:288] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:07.094539 1624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:07.094746 1624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [term 1 LEADER]: Becoming Leader. State: Replica: 650506d148f441a987cdb3a00e87ee79, State: Running, Role: LEADER
I20260926 05:51:07.094777 1614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:07.094887 1624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650506d148f441a987cdb3a00e87ee79" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44467 } }
I20260926 05:51:07.095351 1626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650506d148f441a987cdb3a00e87ee79. Latest consensus state: current_term: 1 leader_uuid: "650506d148f441a987cdb3a00e87ee79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650506d148f441a987cdb3a00e87ee79" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44467 } } }
I20260926 05:51:07.095340 1625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "650506d148f441a987cdb3a00e87ee79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650506d148f441a987cdb3a00e87ee79" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 44467 } } }
I20260926 05:51:07.095454 1626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:07.095463 1625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:07.095887 1633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:07.096403 1633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:07.097929 1633 catalog_manager.cc:1347] Generated new cluster ID: c70ca768789449fe92c4d66ca938023d
I20260926 05:51:07.097982 1633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:07.108491 1633 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:07.108719 1633 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:07.112759 1633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 650506d148f441a987cdb3a00e87ee79: Generated new TSK 0
I20260926 05:51:07.112970 1633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:07.115995 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1545
2026-09-26T05:51:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:07 dist-test-slave-21p4 krb5kdc[1645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:07 dist-test-slave-21p4 krb5kdc[1645](info): set up 2 sockets
Sep 26 05:51:07 dist-test-slave-21p4 krb5kdc[1645](info): commencing operation
krb5kdc: starting...
W20260926 05:51:08.162933 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:08 dist-test-slave-21p4 krb5kdc[1645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:08.289383 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39337
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:08.371546 1658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:08.371814 1658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:08.371840 1658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:08.373006 1658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:08.373046 1658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:08.373060 1658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:08.373073 1658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:08.373085 1658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:08.374541 1658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:08.374769 1658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:08.374993 1658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:08.379766 1664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:08.379789 1663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:08.379935 1658 server_base.cc:1034] running on GCE node
W20260926 05:51:08.379779 1666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:08.380260 1658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:08.380577 1658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:08.381732 1658 hybrid_clock.cc:648] HybridClock initialized: now 1790401868381705 us; error 53 us; skew 500 ppm
Sep 26 05:51:08 dist-test-slave-21p4 krb5kdc[1645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401868, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:08.384531 1658 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:08.385064 1658 webserver.cc:460] Webserver started at http://127.11.247.62:38819/ using document root <none> and password file <none>
I20260926 05:51:08.385313 1658 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:08.385358 1658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:08.385489 1658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:08.386559 1658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "41889c59fe5d4c4fac710eb5fc7925bc"
format_stamp: "Formatted at 2026-09-26 05:51:08 on dist-test-slave-21p4"
server_key: "2f46bd74920459e82c113a4ef1a325db"
server_key_iv: "3ad9ef4b53ed8912daf75dcc1eb88567"
server_key_version: "encryptionkey@0"
I20260926 05:51:08.386988 1658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "41889c59fe5d4c4fac710eb5fc7925bc"
format_stamp: "Formatted at 2026-09-26 05:51:08 on dist-test-slave-21p4"
server_key: "2f46bd74920459e82c113a4ef1a325db"
server_key_iv: "3ad9ef4b53ed8912daf75dcc1eb88567"
server_key_version: "encryptionkey@0"
I20260926 05:51:08.388342 1658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:08.389127 1673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:08.389345 1658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:08.389442 1658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "41889c59fe5d4c4fac710eb5fc7925bc"
format_stamp: "Formatted at 2026-09-26 05:51:08 on dist-test-slave-21p4"
server_key: "2f46bd74920459e82c113a4ef1a325db"
server_key_iv: "3ad9ef4b53ed8912daf75dcc1eb88567"
server_key_version: "encryptionkey@0"
I20260926 05:51:08.389518 1658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:08.398504 1658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:08.400538 1658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:08.400695 1658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:08.404953 1658 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:39337
I20260926 05:51:08.405025 1725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:39337 every 8 connection(s)
I20260926 05:51:08.405354 1658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:08.406448 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:08.408882 1726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc: Bootstrap starting.
I20260926 05:51:08.409521 1726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:08.409829 1726 log.cc:824] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:08.410579 1726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc: No bootstrap required, opened a new log
I20260926 05:51:08.412330 1726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39337 } }
I20260926 05:51:08.412482 1726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:08.412506 1726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41889c59fe5d4c4fac710eb5fc7925bc, State: Initialized, Role: FOLLOWER
I20260926 05:51:08.412678 1726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39337 } }
I20260926 05:51:08.412752 1726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:08.412775 1726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:08.412843 1726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:08.413743 1726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39337 } }
I20260926 05:51:08.413827 1726 leader_election.cc:302] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41889c59fe5d4c4fac710eb5fc7925bc; no voters:
I20260926 05:51:08.414018 1726 leader_election.cc:288] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:08.414079 1729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:08.414301 1726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:08.414332 1729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [term 1 LEADER]: Becoming Leader. State: Replica: 41889c59fe5d4c4fac710eb5fc7925bc, State: Running, Role: LEADER
I20260926 05:51:08.414498 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1658
I20260926 05:51:08.414557 1729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39337 } }
I20260926 05:51:08.414752 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:08.414994 12252 external_mini_cluster.cc:1428] Setting key 056c975eb82e73c2063b1064db890ff1
I20260926 05:51:08.415019 1730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39337 } } }
I20260926 05:51:08.415128 1730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:08.415004 1731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41889c59fe5d4c4fac710eb5fc7925bc. Latest consensus state: current_term: 1 leader_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41889c59fe5d4c4fac710eb5fc7925bc" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39337 } } }
I20260926 05:51:08.415330 1731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:08.415445 1738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:08.415946 1738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:08.417748 1738 catalog_manager.cc:1347] Generated new cluster ID: 74c6b8fbc2844f14a4e15c782021f852
I20260926 05:51:08.417806 1738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:51:08 dist-test-slave-21p4 krb5kdc[1645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:08.425897 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:08.415707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53865 (local address 127.11.247.62:39337)
0926 05:51:08.415974 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:08.415984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:08.416021 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:51:08.416142 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:08.416149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:08.416222 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:08.416390 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:08.417181 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:08.418375 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:08.419234 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:08.419663 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:08.419793 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:08.422217 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:08.422221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:08.422222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:08.422235 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:08.424205 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:08.424582 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:08.424586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:08.424586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:08.424699 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:08.424910 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:08.424911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:08.424911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:08.425169 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:08.425330 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:08.425444 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 05:51:08.425706 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":67,"threads_started":1}
I20260926 05:51:08.426844 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:08.440656 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:08.428206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58543 (local address 127.11.247.62:39337)
0926 05:51:08.428245 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:08.428246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:08.434083 (+ 5837us) server_negotiation.cc:407] Connection header received
0926 05:51:08.434165 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:08.434166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:08.434177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:08.434233 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:08.434867 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:08.435470 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:08.436187 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:08.436428 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:08.436487 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:08.438021 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:08.438023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:08.438023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:08.438030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:08.439505 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:08.439862 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:08.439863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:08.439863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:08.439925 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:08.440120 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:08.440121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:08.440121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:08.440236 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:08.440353 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:08.440456 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 05:51:08.440535 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:08.451535 1738 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:08.452098 1738 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:08.458835 1738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41889c59fe5d4c4fac710eb5fc7925bc: Generated new TSK 0
I20260926 05:51:08.459048 1738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:08.466246 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1658
2026-09-26T05:51:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:08 dist-test-slave-21p4 krb5kdc[1757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:08 dist-test-slave-21p4 krb5kdc[1757](info): set up 2 sockets
Sep 26 05:51:08 dist-test-slave-21p4 krb5kdc[1757](info): commencing operation
krb5kdc: starting...
W20260926 05:51:09.499054 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:09 dist-test-slave-21p4 krb5kdc[1757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:09.629493 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39283
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:39283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:09.715076 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:09.715307 1770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:09.715332 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:09.716553 1770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:09.716609 1770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:09.716624 1770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:09.716636 1770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:09.716648 1770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:09.718205 1770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:39283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:39283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:09.718508 1770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:09.718793 1770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:09.723802 1778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:09.723802 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:09.723935 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:09.724154 1770 server_base.cc:1034] running on GCE node
I20260926 05:51:09.724452 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:09.724763 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:09.725910 1770 hybrid_clock.cc:648] HybridClock initialized: now 1790401869725861 us; error 62 us; skew 500 ppm
Sep 26 05:51:09 dist-test-slave-21p4 krb5kdc[1757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401869, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:09.728590 1770 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:09.729110 1770 webserver.cc:460] Webserver started at http://127.11.247.62:45929/ using document root <none> and password file <none>
I20260926 05:51:09.729353 1770 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:09.729422 1770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:09.729583 1770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:09.730756 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "f441987f0a5f4576ab6a3365c4d069ea"
format_stamp: "Formatted at 2026-09-26 05:51:09 on dist-test-slave-21p4"
server_key: "883e78ea44b4d057a47f742292b259b4"
server_key_iv: "164b44507686031843983319760e6a4f"
server_key_version: "encryptionkey@0"
I20260926 05:51:09.731129 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "f441987f0a5f4576ab6a3365c4d069ea"
format_stamp: "Formatted at 2026-09-26 05:51:09 on dist-test-slave-21p4"
server_key: "883e78ea44b4d057a47f742292b259b4"
server_key_iv: "164b44507686031843983319760e6a4f"
server_key_version: "encryptionkey@0"
I20260926 05:51:09.732533 1770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:51:09.733404 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:09.733659 1770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:09.733793 1770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "f441987f0a5f4576ab6a3365c4d069ea"
format_stamp: "Formatted at 2026-09-26 05:51:09 on dist-test-slave-21p4"
server_key: "883e78ea44b4d057a47f742292b259b4"
server_key_iv: "164b44507686031843983319760e6a4f"
server_key_version: "encryptionkey@0"
I20260926 05:51:09.733884 1770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:09.765695 1770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:09.768132 1770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:09.768328 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:09.772656 1770 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:39283
I20260926 05:51:09.772692 1837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:39283 every 8 connection(s)
I20260926 05:51:09.773068 1770 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:09.774048 1838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:09.775321 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1770
I20260926 05:51:09.775449 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:09.775648 12252 external_mini_cluster.cc:1428] Setting key a21452c06e9efa7d8e555e08b898739e
I20260926 05:51:09.777406 1838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea: Bootstrap starting.
I20260926 05:51:09.778134 1838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:09.778473 1838 log.cc:824] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:09.779381 1838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea: No bootstrap required, opened a new log
I20260926 05:51:09.782079 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:09.776363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38337 (local address 127.11.247.62:39283)
0926 05:51:09.776562 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:09.776575 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:09.776601 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:51:09.777187 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:09.777194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:09.777243 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:09.777383 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:09.777661 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:09.778766 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:09.779965 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:09.780389 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:09.780478 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:09.781347 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:09.781351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:09.781351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:09.781537 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:09.781569 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:09.781581 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:09.781638 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:09.781765 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 05:51:09.781824 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260926 05:51:09.782138 1838 raft_consensus.cc:348] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f441987f0a5f4576ab6a3365c4d069ea" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39283 } }
I20260926 05:51:09.782384 1838 raft_consensus.cc:374] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:09.782464 1838 raft_consensus.cc:729] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f441987f0a5f4576ab6a3365c4d069ea, State: Initialized, Role: FOLLOWER
I20260926 05:51:09.782631 1838 consensus_queue.cc:260] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f441987f0a5f4576ab6a3365c4d069ea" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39283 } }
I20260926 05:51:09.782722 1838 raft_consensus.cc:388] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:09.782781 1838 raft_consensus.cc:482] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:09.782843 1838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:09.783300 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:09.783695 1838 raft_consensus.cc:504] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f441987f0a5f4576ab6a3365c4d069ea" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39283 } }
I20260926 05:51:09.783846 1838 leader_election.cc:302] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f441987f0a5f4576ab6a3365c4d069ea; no voters:
I20260926 05:51:09.784070 1838 leader_election.cc:288] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:09.784102 1843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:09.784224 1843 raft_consensus.cc:686] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [term 1 LEADER]: Becoming Leader. State: Replica: f441987f0a5f4576ab6a3365c4d069ea, State: Running, Role: LEADER
I20260926 05:51:09.784371 1838 sys_catalog.cc:564] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:09.784387 1843 consensus_queue.cc:237] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f441987f0a5f4576ab6a3365c4d069ea" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39283 } }
I20260926 05:51:09.784955 1844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f441987f0a5f4576ab6a3365c4d069ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f441987f0a5f4576ab6a3365c4d069ea" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39283 } } }
I20260926 05:51:09.784978 1845 sys_catalog.cc:455] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader f441987f0a5f4576ab6a3365c4d069ea. Latest consensus state: current_term: 1 leader_uuid: "f441987f0a5f4576ab6a3365c4d069ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f441987f0a5f4576ab6a3365c4d069ea" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 39283 } } }
I20260926 05:51:09.785044 1844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:09.785077 1845 sys_catalog.cc:458] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:09.785627 1852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:09.786339 1852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:09.788278 1852 catalog_manager.cc:1347] Generated new cluster ID: 0c7af7edd8214cc798fbb6172ba7556f
I20260926 05:51:09.788373 1852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:09.788774 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:09.784781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40591 (local address 127.11.247.62:39283)
0926 05:51:09.784821 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:09.784823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:09.785166 (+ 343us) server_negotiation.cc:407] Connection header received
0926 05:51:09.785188 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:09.785189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:09.785200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:09.785258 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:09.785451 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:09.786175 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:09.787284 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:09.787571 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:09.787667 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:09.788495 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:09.788496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:09.788496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:09.788501 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:09.788515 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:09.788520 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:09.788582 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:09.788660 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:51:09.788685 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:09.836246 1852 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:09.837029 1852 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:09.846382 1852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f441987f0a5f4576ab6a3365c4d069ea: Generated new TSK 0
I20260926 05:51:09.846649 1852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:09.861052 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1770
2026-09-26T05:51:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:09 dist-test-slave-21p4 krb5kdc[1869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:09 dist-test-slave-21p4 krb5kdc[1869](info): set up 2 sockets
Sep 26 05:51:09 dist-test-slave-21p4 krb5kdc[1869](info): commencing operation
krb5kdc: starting...
W20260926 05:51:10.898901 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:10 dist-test-slave-21p4 krb5kdc[1869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:11.023037 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:43093
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:43093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:11.105197 1882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:11.105432 1882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:11.105456 1882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:11.106802 1882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:11.106849 1882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:11.106864 1882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:11.106877 1882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:11.106890 1882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:11.108565 1882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:43093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:11.108844 1882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:11.109087 1882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:11.114111 1888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:11.114133 1890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:11.114133 1887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:11.114491 1882 server_base.cc:1034] running on GCE node
I20260926 05:51:11.114797 1882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:11.115082 1882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:11.116240 1882 hybrid_clock.cc:648] HybridClock initialized: now 1790401871116225 us; error 35 us; skew 500 ppm
Sep 26 05:51:11 dist-test-slave-21p4 krb5kdc[1869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401871, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:11.118911 1882 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:11.119407 1882 webserver.cc:460] Webserver started at http://127.11.247.62:40989/ using document root <none> and password file <none>
I20260926 05:51:11.119660 1882 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:11.119707 1882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:11.119828 1882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:11.120927 1882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "78abee9ea71c428c86c0213a59f263e5"
format_stamp: "Formatted at 2026-09-26 05:51:11 on dist-test-slave-21p4"
server_key: "54258c85ccb4dd525d4c46638ce1d92a"
server_key_iv: "3d3a964e679121d41297f5a86f2460ef"
server_key_version: "encryptionkey@0"
I20260926 05:51:11.121310 1882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "78abee9ea71c428c86c0213a59f263e5"
format_stamp: "Formatted at 2026-09-26 05:51:11 on dist-test-slave-21p4"
server_key: "54258c85ccb4dd525d4c46638ce1d92a"
server_key_iv: "3d3a964e679121d41297f5a86f2460ef"
server_key_version: "encryptionkey@0"
I20260926 05:51:11.122848 1882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 05:51:11.123677 1897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:11.123891 1882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:11.123998 1882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "78abee9ea71c428c86c0213a59f263e5"
format_stamp: "Formatted at 2026-09-26 05:51:11 on dist-test-slave-21p4"
server_key: "54258c85ccb4dd525d4c46638ce1d92a"
server_key_iv: "3d3a964e679121d41297f5a86f2460ef"
server_key_version: "encryptionkey@0"
I20260926 05:51:11.124078 1882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:11.127550 1882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:11.129946 1882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:11.130115 1882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:11.134117 1882 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:43093
I20260926 05:51:11.134163 1949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:43093 every 8 connection(s)
I20260926 05:51:11.134536 1882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:11.135600 1950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:11.138077 1950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5: Bootstrap starting.
I20260926 05:51:11.138253 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1882
I20260926 05:51:11.138393 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:11.138710 12252 external_mini_cluster.cc:1428] Setting key 7e0fa6afe69ef77877666c49a6cbf300
I20260926 05:51:11.139029 1950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:11.139441 1950 log.cc:824] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:11.140206 1950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5: No bootstrap required, opened a new log
I20260926 05:51:11.141367 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:11.139251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33643 (local address 127.11.247.62:43093)
0926 05:51:11.139475 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:11.139484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:11.139520 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:51:11.140001 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:11.140008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:11.140035 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:11.140194 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:11.140949 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:11.140953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:11.140954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:11.140958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:11.141003 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:11.141014 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:11.141104 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:11.141138 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 05:51:11.141181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260926 05:51:11.142861 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:11.143503 1950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78abee9ea71c428c86c0213a59f263e5" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43093 } }
I20260926 05:51:11.143719 1950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:51:11.143778 1950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abee9ea71c428c86c0213a59f263e5, State: Initialized, Role: FOLLOWER
I20260926 05:51:11.143934 1950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78abee9ea71c428c86c0213a59f263e5" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43093 } }
I20260926 05:51:11.144052 1950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:51:11.144097 1950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:51:11.144150 1950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:11.144888 1950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78abee9ea71c428c86c0213a59f263e5" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43093 } }
I20260926 05:51:11.145030 1950 leader_election.cc:302] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78abee9ea71c428c86c0213a59f263e5; no voters:
I20260926 05:51:11.145195 1950 leader_election.cc:288] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:51:11.145263 1960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:11.145478 1950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:11.145609 1960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [term 1 LEADER]: Becoming Leader. State: Replica: 78abee9ea71c428c86c0213a59f263e5, State: Running, Role: LEADER
I20260926 05:51:11.145723 1960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78abee9ea71c428c86c0213a59f263e5" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43093 } }
I20260926 05:51:11.145777 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:11.144451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57667 (local address 127.11.247.62:43093)
0926 05:51:11.144487 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:11.144489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:11.144620 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:51:11.144701 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:11.144702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:11.144707 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:11.144763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:11.145448 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:11.145450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:11.145450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:51:11.145455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:11.145471 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:51:11.145476 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:11.145530 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:11.145674 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:51:11.145689 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 05:51:11.146104 1960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78abee9ea71c428c86c0213a59f263e5. Latest consensus state: current_term: 1 leader_uuid: "78abee9ea71c428c86c0213a59f263e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78abee9ea71c428c86c0213a59f263e5" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43093 } } }
I20260926 05:51:11.146180 1960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:11.146377 1969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:11.146706 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78abee9ea71c428c86c0213a59f263e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78abee9ea71c428c86c0213a59f263e5" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 43093 } } }
I20260926 05:51:11.146857 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:11.147104 1969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:11.148808 1969 catalog_manager.cc:1347] Generated new cluster ID: eb44f6b940a34452a891176ce00e44bf
I20260926 05:51:11.148876 1969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:11.161257 1969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:11.161468 1969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:11.170476 1969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78abee9ea71c428c86c0213a59f263e5: Generated new TSK 0
I20260926 05:51:11.170733 1969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:11.176679 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1882
2026-09-26T05:51:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1315 ms)
[----------] 18 tests from AuthTokenIssuingTest (24174 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:11 dist-test-slave-21p4 krb5kdc[1981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:11 dist-test-slave-21p4 krb5kdc[1981](info): set up 2 sockets
Sep 26 05:51:11 dist-test-slave-21p4 krb5kdc[1981](info): commencing operation
krb5kdc: starting...
W20260926 05:51:12.290948 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:12.413908 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37641
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37641,127.11.247.61:38233,127.11.247.60:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:12.495416 1994 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:51:12.495688 1994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:12.495718 1994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:12.495764 1994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:12.496907 1994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:12.496981 1994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:12.497007 1994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:12.497035 1994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:12.497066 1994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:12.498646 1994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37641,127.11.247.61:38233,127.11.247.60:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:12.498961 1994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:12.499223 1994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:12.504318 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:12.504365 1994 server_base.cc:1034] running on GCE node
W20260926 05:51:12.504401 1999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:12.504333 2000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:12.504837 1994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:12.505213 1994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:12.506388 1994 hybrid_clock.cc:648] HybridClock initialized: now 1790401872506377 us; error 50 us; skew 500 ppm
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:12.516134 1994 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:12.516749 1994 webserver.cc:460] Webserver started at http://127.11.247.62:40419/ using document root <none> and password file <none>
I20260926 05:51:12.517021 1994 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:12.517094 1994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:12.517251 1994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:12.518463 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "70bf30fd6c254b2d8f4139409f96a9ef"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "af9f6e71b3832992392c10230ddc9309"
server_key_iv: "267b0b104214fde05e89b4ad1ce28091"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.518913 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "70bf30fd6c254b2d8f4139409f96a9ef"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "af9f6e71b3832992392c10230ddc9309"
server_key_iv: "267b0b104214fde05e89b4ad1ce28091"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.520323 1994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:12.521243 2009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:12.521474 1994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:51:12.521589 1994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "70bf30fd6c254b2d8f4139409f96a9ef"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "af9f6e71b3832992392c10230ddc9309"
server_key_iv: "267b0b104214fde05e89b4ad1ce28091"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.521672 1994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:12.542167 1994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:12.544629 1994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:12.544813 1994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:12.549811 1994 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:37641
I20260926 05:51:12.549868 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:37641 every 8 connection(s)
I20260926 05:51:12.550251 1994 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:12.551419 2062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:12.553160 2062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:12.559104 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 1994
I20260926 05:51:12.559209 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:12.559441 12252 external_mini_cluster.cc:1428] Setting key 85b5445b99a903b813063a0927f6b923
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:12.564158 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.554008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.11.247.62:37641)
0926 05:51:12.554179 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.554188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.554364 (+ 176us) server_negotiation.cc:407] Connection header received
0926 05:51:12.555027 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.555029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.555047 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.555128 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.555682 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.556518 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.557518 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.557904 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.558003 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.560437 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.560445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.560445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.560452 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.562106 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.562813 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.562816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.562817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.562911 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.563228 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.563229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.563230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.563484 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.563559 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.563863 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.563983 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260926 05:51:12.564260 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.553970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37641 (local address 127.0.0.1:33672)
0926 05:51:12.554235 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.554270 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.554863 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.555135 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.555137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.555420 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.555595 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.555599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.556631 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.556632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.557408 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.557411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.558014 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.558015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.558077 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.558716 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.558725 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.560325 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.562241 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.562248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.562251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.562524 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.562995 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.562996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.562996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.563128 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.563571 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.563580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.563669 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.563796 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.564097 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":98,"thread_start_us":114,"threads_started":1}
I20260926 05:51:12.565275 2062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:12.565702 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.565545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36738)
0926 05:51:12.565604 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.565664 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:51:12.565866 2011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
I20260926 05:51:12.566370 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.566266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36754)
0926 05:51:12.566313 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.566346 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:51:12.566522 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:38233: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.11.247.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab.
Entry for principal kudu/127.11.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab.
I20260926 05:51:12.587991 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:38233
--webserver_interface=127.11.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab
--principal=kudu/127.11.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37641,127.11.247.61:38233,127.11.247.60:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:12.615806 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } attempt: 1
I20260926 05:51:12.616374 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.616185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36758)
0926 05:51:12.616271 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.616328 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 05:51:12.616873 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.616675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36764)
0926 05:51:12.616813 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.616848 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260926 05:51:12.617015 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:38233: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
W20260926 05:51:12.674750 2069 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:51:12.674969 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:12.674988 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:12.675010 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:12.676211 2069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:12.676254 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:12.676267 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:12.676280 2069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:12.676292 2069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:12.677888 2069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37641,127.11.247.61:38233,127.11.247.60:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab
--principal=kudu/127.11.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:38233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:12.678181 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:12.678423 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:12.683521 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:12.683521 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:12.683532 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:12.684129 2069 server_base.cc:1034] running on GCE node
I20260926 05:51:12.684401 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:12.684700 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:12.685175 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } attempt: 2
I20260926 05:51:12.685544 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.685430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36768)
0926 05:51:12.685485 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.685517 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:51:12.685878 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.685785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36780)
0926 05:51:12.685840 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.685863 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 05:51:12.685873 2069 hybrid_clock.cc:648] HybridClock initialized: now 1790401872685855 us; error 45 us; skew 500 ppm
W20260926 05:51:12.685968 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:38233: Network error: Client connection negotiation failed: client connection to 127.11.247.61:38233: connect: Connection refused (error 111)
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:12.688588 2069 init.cc:377] Logged in from keytab as kudu/127.11.247.61@KRBTEST.COM (short username kudu)
I20260926 05:51:12.689074 2069 webserver.cc:460] Webserver started at http://127.11.247.61:36209/ using document root <none> and password file <none>
I20260926 05:51:12.689332 2069 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:12.689404 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:12.689551 2069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:12.690711 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data/instance:
uuid: "aea30e6ebc034c57a72a7b2213e4c971"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "9d6af1df38a4a3f7e6db19f5cb257feb"
server_key_iv: "7c70c3f4dcffeb2c7821d12d6390f04c"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.691061 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal/instance:
uuid: "aea30e6ebc034c57a72a7b2213e4c971"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "9d6af1df38a4a3f7e6db19f5cb257feb"
server_key_iv: "7c70c3f4dcffeb2c7821d12d6390f04c"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.692435 2069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:51:12.693244 2084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:12.693429 2069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:12.693535 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal
uuid: "aea30e6ebc034c57a72a7b2213e4c971"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "9d6af1df38a4a3f7e6db19f5cb257feb"
server_key_iv: "7c70c3f4dcffeb2c7821d12d6390f04c"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.693614 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:12.720620 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:12.723055 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:12.723232 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:12.727670 2069 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.61:38233
I20260926 05:51:12.727715 2136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.61:38233 every 8 connection(s)
I20260926 05:51:12.728078 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
I20260926 05:51:12.729143 2137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:12.730808 2137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:12.733924 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 2069
I20260926 05:51:12.734022 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-1/wal/instance
I20260926 05:51:12.734221 12252 external_mini_cluster.cc:1428] Setting key b740dbf5128e89ddccf133dfe10f55c1
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:12.741739 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.731584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.11.247.62:37641)
0926 05:51:12.731681 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.731684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.731896 (+ 212us) server_negotiation.cc:407] Connection header received
0926 05:51:12.732370 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.732371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.732386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.732453 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.733193 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.733784 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.734998 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.735360 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.735443 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.738341 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.738344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.738344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.738354 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.740189 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.740651 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.740652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.740653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.740724 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.740994 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.740995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.740996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.741169 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.741259 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.741520 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.741610 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 05:51:12.741868 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.731545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37641 (local address 127.0.0.1:33674)
0926 05:51:12.731746 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.731791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.732280 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.732535 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.732538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.732852 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.733056 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.733060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.733893 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.733895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.734908 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.734910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.735518 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.735520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.735760 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.736336 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.736348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.738230 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.740300 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.740308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.740310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.740571 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.740803 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.740804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.740804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.740918 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.741243 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.741248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.741384 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.741472 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.741604 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260926 05:51:12.742573 2137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.247.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.60@KRBTEST.COM" created.
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:51:12.751127 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.742979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36788)
0926 05:51:12.743025 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.743038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.743086 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.743376 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.743377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.743493 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.743547 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.743548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.744499 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.744500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.745056 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.745058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.745549 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.745550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.745645 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.746373 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.746378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.747888 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.749893 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.749894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.749895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.750065 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.750297 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.750298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.750298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.750353 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.750689 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.750690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.750748 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.750829 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.751029 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":99}
I20260926 05:51:12.751130 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.743036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.11.247.61:38233)
0926 05:51:12.743224 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.743229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.743249 (+ 20us) server_negotiation.cc:407] Connection header received
0926 05:51:12.743279 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.743281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.743292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.743354 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.743657 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.744406 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.745131 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.745465 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.745527 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.747997 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.748008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.748009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.748016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.749776 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.750159 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.750162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.750163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.750229 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.750420 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.750421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.750421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.750633 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.750713 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.750862 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.750977 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260926 05:51:12.751979 2137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:12.752360 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.752223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49390)
0926 05:51:12.752286 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.752338 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:51:12.752504 2087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
I20260926 05:51:12.752909 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.752820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49394)
0926 05:51:12.752863 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.752887 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:51:12.753026 2137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34313: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab.
Entry for principal kudu/127.11.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab.
I20260926 05:51:12.762107 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:34313
--webserver_interface=127.11.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab
--principal=kudu/127.11.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:37641,127.11.247.61:38233,127.11.247.60:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:12.777130 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } attempt: 3
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:51:12.784531 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.777642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.11.247.61:38233)
0926 05:51:12.777716 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.777719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.777731 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:12.777795 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.777796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.777816 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.777879 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.778223 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.778836 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.779578 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.779799 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.779885 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.781940 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.781941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.781942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.781950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.783420 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.783810 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.783811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.783811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.783859 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.784047 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.784047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.784047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.784153 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.784208 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.784377 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.784463 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:51:12.784556 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.777570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36792)
0926 05:51:12.777636 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.777651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.777722 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.777893 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.777894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.778063 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.778131 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.778133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.778943 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.778944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.779489 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.779490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.779888 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.779889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.779980 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.780475 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.780480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.781817 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.783509 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.783510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.783511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.783737 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.783920 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.783921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.783922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.783988 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.784216 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.784217 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.784281 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.784353 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.784458 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:51:12.785063 2062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:12.785429 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.785314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49400)
0926 05:51:12.785367 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.785403 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:51:12.785833 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.785726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49414)
0926 05:51:12.785789 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.785815 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 05:51:12.785949 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34313: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
I20260926 05:51:12.802281 2137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } attempt: 1
I20260926 05:51:12.802881 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.802733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49418)
0926 05:51:12.802804 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.802848 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 05:51:12.803311 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.803176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49432)
0926 05:51:12.803254 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.803286 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 05:51:12.803435 2137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34313: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
I20260926 05:51:12.817112 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } attempt: 1
I20260926 05:51:12.817672 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.817519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49434)
0926 05:51:12.817583 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.817634 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:51:12.818038 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.817947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49436)
0926 05:51:12.817988 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.818017 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:51:12.818133 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34313: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
W20260926 05:51:12.849156 2144 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:51:12.849408 2144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:12.849428 2144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:12.849452 2144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:12.850651 2144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:12.850692 2144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:12.850713 2144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:12.850725 2144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:12.850737 2144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:12.852257 2144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:37641,127.11.247.61:38233,127.11.247.60:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab
--principal=kudu/127.11.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:12.852478 2144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:12.852662 2144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:12.857896 2149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:12.857900 2152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:12.857890 2150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:12.858170 2144 server_base.cc:1034] running on GCE node
I20260926 05:51:12.858438 2144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:12.858820 2144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:12.860005 2144 hybrid_clock.cc:648] HybridClock initialized: now 1790401872859978 us; error 49 us; skew 500 ppm
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:12.862739 2144 init.cc:377] Logged in from keytab as kudu/127.11.247.60@KRBTEST.COM (short username kudu)
I20260926 05:51:12.863260 2144 webserver.cc:460] Webserver started at http://127.11.247.60:41639/ using document root <none> and password file <none>
I20260926 05:51:12.863531 2144 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:12.863603 2144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:12.863761 2144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:12.864912 2144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data/instance:
uuid: "a70ca64003134b93a1c05584ed83f202"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "1e850bdbab92d812363c6cb6151909bd"
server_key_iv: "a0e971823ba0a9f01d1ba0597cbbd99e"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.865296 2144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal/instance:
uuid: "a70ca64003134b93a1c05584ed83f202"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "1e850bdbab92d812363c6cb6151909bd"
server_key_iv: "a0e971823ba0a9f01d1ba0597cbbd99e"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.866732 2144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:12.867579 2159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:12.867760 2144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:51:12.867873 2144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal
uuid: "a70ca64003134b93a1c05584ed83f202"
format_stamp: "Formatted at 2026-09-26 05:51:12 on dist-test-slave-21p4"
server_key: "1e850bdbab92d812363c6cb6151909bd"
server_key_iv: "a0e971823ba0a9f01d1ba0597cbbd99e"
server_key_version: "encryptionkey@0"
I20260926 05:51:12.867950 2144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:12.871567 2137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } attempt: 2
I20260926 05:51:12.871999 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.871866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49450)
0926 05:51:12.871931 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.871971 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 05:51:12.872351 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.872243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49452)
0926 05:51:12.872286 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.872310 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:51:12.872432 2137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34313: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
I20260926 05:51:12.887552 2144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:12.889856 2144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:12.890038 2144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:12.893277 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } attempt: 2
I20260926 05:51:12.893949 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.893660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49468)
0926 05:51:12.893829 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.893909 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 05:51:12.894392 2144 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.60:34313
I20260926 05:51:12.894434 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.894313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49470)
0926 05:51:12.894349 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.894401 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 05:51:12.894454 2211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.60:34313 every 8 connection(s)
W20260926 05:51:12.894536 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34313: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34313: connect: Connection refused (error 111)
I20260926 05:51:12.894914 2144 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
I20260926 05:51:12.895957 2212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:12.897688 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 2144
I20260926 05:51:12.897783 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401595249435-12252-0/minicluster-data/master-2/wal/instance
I20260926 05:51:12.898006 12252 external_mini_cluster.cc:1428] Setting key 34af21f181b8f2381c16469c3f332397
I20260926 05:51:12.897958 2212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:51:12.908104 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.898888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.11.247.62:37641)
0926 05:51:12.898934 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.898936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.899126 (+ 190us) server_negotiation.cc:407] Connection header received
0926 05:51:12.899595 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.899597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.899610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.899679 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.900322 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.900835 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.901745 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.901959 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.902049 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.904699 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.904700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.904701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.904717 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.906319 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.906879 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.906880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.906881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.906947 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.907292 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.907294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.907295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.907444 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.907528 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.907892 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.907985 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:12.908394 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.898717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.11.247.60:34313)
0926 05:51:12.898909 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.898919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.898958 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:51:12.899435 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.899442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.899464 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.899599 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.900306 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.901206 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.901989 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.902300 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.902382 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.905036 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.905045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.905045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.905051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.907124 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.907492 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.907496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.907496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.907558 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.907763 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.907771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.907772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.907925 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.908117 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.908162 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.908244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260926 05:51:12.908478 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.898844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:37641 (local address 127.0.0.1:33678)
0926 05:51:12.899000 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.899043 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.899462 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.899702 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.899706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.900036 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.900207 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.900211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.900916 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.900918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.901640 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.901641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.902088 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.902089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.902209 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.902789 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.902800 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.904388 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.906445 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.906454 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.906456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.906789 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.907067 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.907068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.907068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.907188 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.907521 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.907529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.907689 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.907859 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.908078 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260926 05:51:12.909090 2212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:12.916858 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.909387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:38233 (local address 127.0.0.1:36808)
0926 05:51:12.909464 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.909476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.909523 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.909689 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.909690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.909854 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.909902 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.909904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.910662 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.910663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.911181 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.911183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.911600 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.911602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.911680 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.912187 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.912193 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.913733 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.915715 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.915717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.915718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.915971 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.916183 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.916184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.916184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.916250 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.916549 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.916550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.916595 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.916650 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.916786 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 05:51:12.916894 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.909531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.11.247.61:38233)
0926 05:51:12.909575 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.909578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.909585 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:12.909609 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.909610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.909619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.909674 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.910037 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.910535 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.911266 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.911508 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.911601 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.913841 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.913843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.913843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.913853 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.915624 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.916048 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.916049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.916049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.916110 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.916314 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.916315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.916316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.916475 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.916554 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.916693 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.916777 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:12.917299 2212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:12.917847 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.909574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.11.247.62:37641)
0926 05:51:12.909614 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.909616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.909625 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:12.909691 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.909692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.909701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.909750 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.910262 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.910835 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.911537 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.911753 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.911832 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.914874 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.914876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.914877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.914887 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.916524 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.916946 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.916947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.916947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.917009 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.917256 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.917257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.917257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.917425 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.917615 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.917653 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.917744 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:51:12.925519 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.917611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49486)
0926 05:51:12.917663 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.917673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.917719 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.917856 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.917857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.918022 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.918063 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.918065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.918868 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.918870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.919659 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.919661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.920068 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.920069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.920112 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.920567 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.920572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.922150 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.924056 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.924057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.924058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.924382 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.924826 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.924827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.924827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.924888 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.925146 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.925147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.925187 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.925277 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.925393 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:51:12.925630 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.917642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.11.247.60:34313)
0926 05:51:12.917675 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.917677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.917704 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:51:12.917763 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.917764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.917773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.917818 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.918131 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.918789 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.919746 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.919988 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.920058 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.922248 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.922250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.922250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.922259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.923967 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.924688 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.924689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.924689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.924748 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.924953 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.924954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.924954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.925078 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.925156 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.925300 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.925469 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":19}
I20260926 05:51:12.927055 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.918549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.11.247.61:38233)
0926 05:51:12.918616 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.918618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.918629 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:12.918650 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.918651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.918659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.918713 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.919229 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.919884 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.920763 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.921078 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.921149 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.923900 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.923901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.923901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.923909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.925797 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.926188 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.926189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.926190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.926284 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.926491 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.926492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.926492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.926672 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.926842 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.926887 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.926965 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:12.927656 2212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: Bootstrap starting.
I20260926 05:51:12.928653 2212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:12.928690 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:51:12.928975 2212 log.cc:824] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:12.929867 2212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: No bootstrap required, opened a new log
I20260926 05:51:12.929916 2036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "70bf30fd6c254b2d8f4139409f96a9ef"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684
I20260926 05:51:12.931991 2212 raft_consensus.cc:348] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:12.932178 2212 raft_consensus.cc:729] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70ca64003134b93a1c05584ed83f202, State: Initialized, Role: FOLLOWER
I20260926 05:51:12.932260 2212 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:12.932627 2212 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:12.932570 2218 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:12.932703 2218 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:51:12.934713 2229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:12.934782 2229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:12.935180 2144 master_runner.cc:186] Error getting master registration for 127.11.247.62:37641: OK, instance_id { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" instance_seqno: 1790401872548261 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:51:12.938187 2144 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:51:12.963585 2137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } attempt: 3
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:51:12.971747 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.964040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49502)
0926 05:51:12.964099 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.964112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.964240 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.964419 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.964420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.964620 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.964673 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.964675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.965367 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.965368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.965880 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.965882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.966268 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.966268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.966325 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.966943 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.966949 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.968485 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:12.970467 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.970468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.970469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.970759 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.970988 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:12.970988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:12.970989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:12.971060 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:12.971376 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:12.971377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:12.971450 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 05:51:12.971512 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:51:12.971627 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 05:51:12.971774 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.964137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.11.247.60:34313)
0926 05:51:12.964272 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.964274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.964287 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:51:12.964347 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.964347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.964361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.964413 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.964768 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.965267 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.965987 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.966191 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.966290 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.968613 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.968615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.968616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.968627 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:12.970369 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.970849 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.970850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.970850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.970917 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:12.971116 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:12.971117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:12.971117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:12.971279 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:12.971373 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:12.971581 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 05:51:12.971671 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 05:51:12.974400 2137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: Bootstrap starting.
I20260926 05:51:12.975369 2137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:12.975695 2137 log.cc:824] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:12.976712 2137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: No bootstrap required, opened a new log
I20260926 05:51:12.978941 2137 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:12.979123 2137 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea30e6ebc034c57a72a7b2213e4c971, State: Initialized, Role: FOLLOWER
I20260926 05:51:12.979223 2137 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:12.979631 2137 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:12.979578 2232 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:12.979707 2232 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:51:12.981623 2243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:12.981688 2243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:12.981902 2069 master_runner.cc:186] Error getting master registration for 127.11.247.62:37641: OK, instance_id { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" instance_seqno: 1790401872548261 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:51:12.984805 2069 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:51:12.993680 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } attempt: 3
Sep 26 05:51:12 dist-test-slave-21p4 krb5kdc[1981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401872, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:51:13.002035 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.994198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.11.247.60:34313)
0926 05:51:12.994241 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:12.994244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:12.994252 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:12.994280 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:12.994281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:12.994291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:12.994351 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:12.994773 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.995525 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.996570 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.996854 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.996922 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.999252 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:12.999254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:12.999254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:12.999263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:13.000740 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.001160 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.001162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.001162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.001223 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.001412 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.001413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.001414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.001601 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:13.001668 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:13.001842 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 05:51:13.001934 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:13.002035 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:12.994063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34313 (local address 127.0.0.1:49510)
0926 05:51:12.994116 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:12.994131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:12.994182 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:12.994370 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:12.994371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:12.994551 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:12.994685 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.994687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.995622 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.995623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.996477 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:12.996480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:12.996928 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:12.996929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:12.996995 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:12.997556 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:12.997562 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:12.999151 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:13.000861 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:13.000862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:13.000863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:13.001099 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:13.001295 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:13.001297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:13.001297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:13.001360 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:13.001669 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:13.001670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:13.001722 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 05:51:13.001816 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:51:13.001928 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 05:51:13.004179 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: Bootstrap starting.
I20260926 05:51:13.004956 2062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:13.005273 2062 log.cc:824] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:13.006171 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: No bootstrap required, opened a new log
I20260926 05:51:13.007836 2062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:13.007999 2062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70bf30fd6c254b2d8f4139409f96a9ef, State: Initialized, Role: FOLLOWER
I20260926 05:51:13.008092 2062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:13.008484 2062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:13.008579 2036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 05:51:13.008631 2036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:13.009608 2036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:13.009902 2036 leader_election.cc:288] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [CANDIDATE]: Term 1 election: Requested vote from peers aea30e6ebc034c57a72a7b2213e4c971 (127.11.247.61:38233), a70ca64003134b93a1c05584ed83f202 (127.11.247.60:34313)
I20260926 05:51:13.010501 2111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aea30e6ebc034c57a72a7b2213e4c971"
I20260926 05:51:13.010749 2111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [term 0 FOLLOWER]: Advancing to term 1
W20260926 05:51:13.010833 2257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:13.010877 2257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:13.011740 2111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70bf30fd6c254b2d8f4139409f96a9ef in term 1.
I20260926 05:51:13.012001 2186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a70ca64003134b93a1c05584ed83f202"
I20260926 05:51:13.012120 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:13.012984 2186 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70bf30fd6c254b2d8f4139409f96a9ef in term 1.
I20260926 05:51:13.014714 2011 leader_election.cc:302] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70bf30fd6c254b2d8f4139409f96a9ef, aea30e6ebc034c57a72a7b2213e4c971; no voters:
I20260926 05:51:13.017534 1994 master_runner.cc:263] No leader master found from master 70bf30fd6c254b2d8f4139409f96a9ef
I20260926 05:51:13.017598 1994 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:51:13.008417 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.017871 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:13.017985 2246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:13.018098 2246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [term 1 LEADER]: Becoming Leader. State: Replica: 70bf30fd6c254b2d8f4139409f96a9ef, State: Running, Role: LEADER
I20260926 05:51:13.018235 2246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } }
I20260926 05:51:13.018855 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70bf30fd6c254b2d8f4139409f96a9ef. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.019012 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:13.020538 2265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:13.021368 2265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:13.031711 2186 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [term 1 FOLLOWER]: Refusing update from remote peer 70bf30fd6c254b2d8f4139409f96a9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:13.032135 2246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:13.032787 2111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [term 1 FOLLOWER]: Refusing update from remote peer 70bf30fd6c254b2d8f4139409f96a9ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:13.033254 2263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:13.034793 2232 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70bf30fd6c254b2d8f4139409f96a9ef. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.034888 2232 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:13.036154 2218 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70bf30fd6c254b2d8f4139409f96a9ef. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.036257 2218 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:13.040539 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:13.019438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.11.247.60:34313)
0926 05:51:13.019485 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.019487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.022290 (+ 2803us) server_negotiation.cc:407] Connection header received
0926 05:51:13.022408 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.022409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.022421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.022454 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.023373 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:13.024142 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:13.027541 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:13.027860 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:13.027902 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:13.037332 (+ 9430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:13.037334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:13.037335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:13.037344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:13.039206 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.039648 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.039649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.039649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.039703 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.039891 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.039891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.039892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.040051 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:13.040132 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:13.040335 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 05:51:13.040432 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:51:13.045691 2015 tablet.cc:2369] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:51:13.048274 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:13.019889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.11.247.61:38233)
0926 05:51:13.019946 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.019948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.025558 (+ 5610us) server_negotiation.cc:407] Connection header received
0926 05:51:13.025694 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.025695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.025708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.025753 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.028501 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:13.029269 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:13.043092 (+ 13823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:13.043429 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:13.043478 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:13.045126 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:13.045128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:13.045128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:13.045136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:13.047000 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.047396 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.047397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.047397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.047454 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.047639 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.047640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.047641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.047773 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:13.047868 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:13.048087 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 05:51:13.048177 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:13.053733 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:13.020272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.11.247.62:37641)
0926 05:51:13.020313 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.020315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.026676 (+ 6361us) server_negotiation.cc:407] Connection header received
0926 05:51:13.026773 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.026774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.026786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.026814 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.030671 (+ 3857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:13.031310 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:13.042140 (+ 10830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:13.042471 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:13.042515 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:13.050642 (+ 8127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:13.050645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:13.050645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:13.050653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:13.052415 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.052823 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.052824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.052824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.052879 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:13.053086 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:13.053086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:13.053087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:13.053228 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:13.053311 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:13.053519 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 05:51:13.053626 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:13.057212 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.057330 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:13.057938 2218 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.058037 2218 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:13.058317 2272 mvcc.cc:205] Tried to move back new op lower bound from 7333486071934906368 to 7333486071884226560. Current Snapshot: MvccSnapshot[applied={T|T < 7333486071934906368}]
I20260926 05:51:13.058823 2263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.058938 2263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:13.059507 2232 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bf30fd6c254b2d8f4139409f96a9ef" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 37641 } } peers { permanent_uuid: "aea30e6ebc034c57a72a7b2213e4c971" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 38233 } } peers { permanent_uuid: "a70ca64003134b93a1c05584ed83f202" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34313 } } }
I20260926 05:51:13.059604 2232 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:13.060860 2270 mvcc.cc:205] Tried to move back new op lower bound from 7333486071934906368 to 7333486071884226560. Current Snapshot: MvccSnapshot[applied={T|T < 7333486071934906368}]
I20260926 05:51:13.063086 2265 catalog_manager.cc:1347] Generated new cluster ID: 5cedaec119014bbb95e2353cf139642d
I20260926 05:51:13.063201 2265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:13.076853 2265 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:13.077715 2265 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:13.088285 2265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70bf30fd6c254b2d8f4139409f96a9ef: Generated new TSK 0
I20260926 05:51:13.089638 2265 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:51:13.120019 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.118676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.11.247.61:38233)
0926 05:51:13.118719 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.118721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.119097 (+ 376us) server_negotiation.cc:407] Connection header received
0926 05:51:13.119321 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.119322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.119337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.119371 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.119936 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36834: BlockingRecv error: recv got EOF from 127.0.0.1:36834 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:51:13.120358 2064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.118910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.11.247.62:37641)
0926 05:51:13.118953 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.118955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.119247 (+ 292us) server_negotiation.cc:407] Connection header received
0926 05:51:13.119895 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.119897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.119910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.119941 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.120268 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33700: BlockingRecv error: recv got EOF from 127.0.0.1:33700 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 05:51:13.120569 2215 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.118521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.11.247.60:34313)
0926 05:51:13.118633 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.118636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.119221 (+ 585us) server_negotiation.cc:407] Connection header received
0926 05:51:13.119360 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.119362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.119375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.119407 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.120505 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49530: BlockingRecv error: recv got EOF from 127.0.0.1:49530 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 05:51:13.122975 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.122182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.11.247.61:38233)
0926 05:51:13.122227 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.122229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.122337 (+ 108us) server_negotiation.cc:407] Connection header received
0926 05:51:13.122413 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.122414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.122438 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.122508 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.122920 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36848: BlockingRecv error: recv got EOF from 127.0.0.1:36848 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:51:13.126037 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.125412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.11.247.61:38233)
0926 05:51:13.125480 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.125482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.125490 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:13.125519 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.125520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.125532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.125599 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.125975 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36854: BlockingRecv error: recv got EOF from 127.0.0.1:36854 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 05:51:13.130329 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.129505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.11.247.61:38233)
0926 05:51:13.129556 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.129559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.129717 (+ 158us) server_negotiation.cc:407] Connection header received
0926 05:51:13.129778 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.129779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.129792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.129863 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.130256 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36864: BlockingRecv error: recv got EOF from 127.0.0.1:36864 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:51:13.136257 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.135478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.11.247.61:38233)
0926 05:51:13.135527 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.135529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.135639 (+ 110us) server_negotiation.cc:407] Connection header received
0926 05:51:13.135707 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.135708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.135723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.135816 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.136196 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36872: BlockingRecv error: recv got EOF from 127.0.0.1:36872 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 05:51:13.146291 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.145632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.11.247.61:38233)
0926 05:51:13.145677 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.145679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.145691 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:13.145755 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.145756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.145768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.145835 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.146224 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36876: BlockingRecv error: recv got EOF from 127.0.0.1:36876 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:51:13.164731 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.163753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.11.247.61:38233)
0926 05:51:13.163829 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.163831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.163927 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:51:13.164022 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.164024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.164041 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.164107 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.164662 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36892: BlockingRecv error: recv got EOF from 127.0.0.1:36892 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 05:51:13.198684 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.197861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.11.247.61:38233)
0926 05:51:13.197948 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.197950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.197999 (+ 49us) server_negotiation.cc:407] Connection header received
0926 05:51:13.198039 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.198040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.198052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.198108 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.198572 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36900: BlockingRecv error: recv got EOF from 127.0.0.1:36900 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 05:51:13.265082 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.264213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.11.247.61:38233)
0926 05:51:13.264303 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.264305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.264356 (+ 51us) server_negotiation.cc:407] Connection header received
0926 05:51:13.264461 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.264462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.264475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.264547 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.265030 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36908: BlockingRecv error: recv got EOF from 127.0.0.1:36908 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 05:51:13.395081 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.394368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.11.247.61:38233)
0926 05:51:13.394424 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.394426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.394468 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:51:13.394548 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.394549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.394560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.394683 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.395030 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36920: BlockingRecv error: recv got EOF from 127.0.0.1:36920 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:51:13.653306 2141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:13.652658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.11.247.61:38233)
0926 05:51:13.652733 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:13.652736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:13.652744 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:13.652774 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:13.652774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:13.652786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:13.652866 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:13.653246 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36932: BlockingRecv error: recv got EOF from 127.0.0.1:36932 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:51:13.935249 2229 catalog_manager.cc:1259] Loaded cluster ID: 5cedaec119014bbb95e2353cf139642d
I20260926 05:51:13.935326 2229 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: loading cluster ID for follower catalog manager: success
I20260926 05:51:13.936177 2229 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: acquiring CA information for follower catalog manager: success
I20260926 05:51:13.936503 2229 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a70ca64003134b93a1c05584ed83f202: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:51:13.982218 2243 catalog_manager.cc:1259] Loaded cluster ID: 5cedaec119014bbb95e2353cf139642d
I20260926 05:51:13.982298 2243 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: loading cluster ID for follower catalog manager: success
I20260926 05:51:13.983232 2243 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: acquiring CA information for follower catalog manager: success
I20260926 05:51:13.983564 2243 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aea30e6ebc034c57a72a7b2213e4c971: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:51:14.170120 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:14.166816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.11.247.61:38233)
0926 05:51:14.166925 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:14.166927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:14.166941 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:51:14.167043 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:14.167044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:14.167097 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:14.167176 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:14.167363 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.167962 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.168917 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.169154 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.169239 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:14.169516 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:14.169709 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:14.169814 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:14.169937 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 05:51:14.169964 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260926 05:51:14.175709 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:14.172608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.11.247.60:34313)
0926 05:51:14.172754 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:14.172757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:14.172817 (+ 60us) server_negotiation.cc:407] Connection header received
0926 05:51:14.172922 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:14.172923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:14.172989 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:14.173099 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:14.173260 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.173807 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.174736 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.174970 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.175062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:14.175273 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:14.175411 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:14.175519 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:14.175563 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 05:51:14.175589 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
I20260926 05:51:14.181116 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:14.177783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.11.247.61:38233)
0926 05:51:14.177877 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:14.177879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:14.178117 (+ 238us) server_negotiation.cc:407] Connection header received
0926 05:51:14.178158 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:14.178159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:14.178173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:14.178266 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:14.178408 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.179240 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.180310 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.180600 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.180671 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:14.180821 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:14.180860 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:14.180926 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:14.180997 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 05:51:14.181029 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 05:51:14.181150 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:14.177807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.11.247.60:34313)
0926 05:51:14.177852 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:14.177854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:14.178142 (+ 288us) server_negotiation.cc:407] Connection header received
0926 05:51:14.178255 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:14.178256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:14.178270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:14.178322 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:14.178493 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.179042 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.180293 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.180593 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.180628 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:14.180852 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:14.180887 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:14.180944 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:14.181054 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 05:51:14.181078 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:51:14.181773 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:14.177855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.11.247.62:37641)
0926 05:51:14.177993 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:14.177995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:14.178005 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:14.178111 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:14.178112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:14.178172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:14.178269 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:14.178475 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.179267 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.180214 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:14.180594 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:14.180671 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:14.181032 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:14.181167 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:14.181233 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:14.181641 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 05:51:14.181674 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260926 05:51:14.183985 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 1994
I20260926 05:51:14.189249 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 2069
I20260926 05:51:14.195288 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 2144
2026-09-26T05:51:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3020 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:51:14 dist-test-slave-21p4 krb5kdc[2358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:51:14 dist-test-slave-21p4 krb5kdc[2358](info): set up 2 sockets
Sep 26 05:51:14 dist-test-slave-21p4 krb5kdc[2358](info): commencing operation
krb5kdc: starting...
W20260926 05:51:15.259279 12252 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/test-user.keytab.
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:51:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.247.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
Entry for principal kudu/127.11.247.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab.
I20260926 05:51:15.387099 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42137
--webserver_interface=127.11.247.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42137,127.11.247.61:34927,127.11.247.60:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:51:15.470960 2371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:51:15.471202 2371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:15.471220 2371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:15.471243 2371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:15.472508 2371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:15.472561 2371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:15.472576 2371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:15.472589 2371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:15.472601 2371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:15.474222 2371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42137,127.11.247.61:34927,127.11.247.60:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.62.keytab
--principal=kudu/127.11.247.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.62:42137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.247.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:15.474510 2371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:15.474817 2371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:15.480096 2376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:15.480113 2371 server_base.cc:1034] running on GCE node
W20260926 05:51:15.480082 2377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:15.480082 2379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:15.480569 2371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:15.480924 2371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:15.482100 2371 hybrid_clock.cc:648] HybridClock initialized: now 1790401875482090 us; error 51 us; skew 500 ppm
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:15.484766 2371 init.cc:377] Logged in from keytab as kudu/127.11.247.62@KRBTEST.COM (short username kudu)
I20260926 05:51:15.485255 2371 webserver.cc:460] Webserver started at http://127.11.247.62:42851/ using document root <none> and password file <none>
I20260926 05:51:15.485513 2371 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:15.485587 2371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:15.485740 2371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:15.486946 2371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data/instance:
uuid: "c0516847e6e24030931ef2cfd29ab480"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "cf42dcc257351ac2886616466621aef1"
server_key_iv: "b3266c77883e9bb6bc174ff65370244a"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.487452 2371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal/instance:
uuid: "c0516847e6e24030931ef2cfd29ab480"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "cf42dcc257351ac2886616466621aef1"
server_key_iv: "b3266c77883e9bb6bc174ff65370244a"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.489030 2371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:51:15.489881 2386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:15.490125 2371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:15.490247 2371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal
uuid: "c0516847e6e24030931ef2cfd29ab480"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "cf42dcc257351ac2886616466621aef1"
server_key_iv: "b3266c77883e9bb6bc174ff65370244a"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.490339 2371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:15.504006 2371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:15.506436 2371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:15.506649 2371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:15.511157 2371 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.62:42137
I20260926 05:51:15.511210 2438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.62:42137 every 8 connection(s)
I20260926 05:51:15.511600 2371 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/data/info.pb
I20260926 05:51:15.512334 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 2371
I20260926 05:51:15.512465 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-0/wal/instance
I20260926 05:51:15.512727 12252 external_mini_cluster.cc:1428] Setting key e568f6e87d1f30e8a24c3c6c4c0b84db
I20260926 05:51:15.513056 2439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:15.515475 2439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.247.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.61@KRBTEST.COM" created.
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:15.530156 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.516292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42137 (local address 127.0.0.1:34210)
0926 05:51:15.516541 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.516603 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.517328 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.517672 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.517676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.518115 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.518387 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.518393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.519532 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.519534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.520761 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.520765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.521301 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.521303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.521407 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.522039 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.522048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.525998 (+ 3950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.528422 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.528446 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.528449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.528667 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.529010 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.529011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.529011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.529123 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.529635 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.529641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.529707 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.529791 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.529967 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":133,"threads_started":1}
I20260926 05:51:15.530248 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.516386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.11.247.62:42137)
0926 05:51:15.516553 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.516562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.516717 (+ 155us) server_negotiation.cc:407] Connection header received
0926 05:51:15.517511 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.517513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.517531 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.517611 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.518498 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.519420 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.520860 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.521181 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.521273 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.526116 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.526126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.526126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.526133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.528313 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.528764 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.528767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.528768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.528901 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.529202 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.529203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.529203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.529554 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.529649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.529861 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.530061 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260926 05:51:15.531311 2439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:15.531807 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.531594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53582)
0926 05:51:15.531672 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.531779 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 05:51:15.532008 2388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
I20260926 05:51:15.532543 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.532417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53598)
0926 05:51:15.532489 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.532522 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 05:51:15.532713 2439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:34927: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab.
Entry for principal kudu/127.11.247.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab.
I20260926 05:51:15.542986 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:34927
--webserver_interface=127.11.247.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab
--principal=kudu/127.11.247.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42137,127.11.247.61:34927,127.11.247.60:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:15.582018 2439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } attempt: 1
I20260926 05:51:15.582638 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.582462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53606)
0926 05:51:15.582533 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.582599 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 05:51:15.583103 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.582938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53616)
0926 05:51:15.583049 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.583081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 05:51:15.583216 2439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:34927: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
W20260926 05:51:15.628520 2446 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:51:15.628762 2446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:15.628780 2446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:15.628810 2446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:15.629954 2446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:15.629997 2446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:15.630010 2446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:15.630023 2446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:15.630035 2446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:15.631531 2446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42137,127.11.247.61:34927,127.11.247.60:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.61.keytab
--principal=kudu/127.11.247.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.61:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.247.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:15.631749 2446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:15.631940 2446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:51:15.637136 2454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:15.637199 2452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:15.637271 2446 server_base.cc:1034] running on GCE node
W20260926 05:51:15.637146 2451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:15.637651 2446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:15.638016 2446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:15.639192 2446 hybrid_clock.cc:648] HybridClock initialized: now 1790401875639177 us; error 30 us; skew 500 ppm
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:15.641698 2446 init.cc:377] Logged in from keytab as kudu/127.11.247.61@KRBTEST.COM (short username kudu)
I20260926 05:51:15.642107 2446 webserver.cc:460] Webserver started at http://127.11.247.61:44115/ using document root <none> and password file <none>
I20260926 05:51:15.642293 2446 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:15.642333 2446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:15.642431 2446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:15.643563 2446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data/instance:
uuid: "13528de440eb41bb9c50425c7c894d73"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "0e9f37fda57702fb34b0af77481fceda"
server_key_iv: "16e6e8c64a7f2926130435d6560b6580"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.643865 2446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal/instance:
uuid: "13528de440eb41bb9c50425c7c894d73"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "0e9f37fda57702fb34b0af77481fceda"
server_key_iv: "16e6e8c64a7f2926130435d6560b6580"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.645172 2446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:51:15.645926 2461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:15.646139 2446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:51:15.646231 2446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal
uuid: "13528de440eb41bb9c50425c7c894d73"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "0e9f37fda57702fb34b0af77481fceda"
server_key_iv: "16e6e8c64a7f2926130435d6560b6580"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.646296 2446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:15.651357 2439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } attempt: 2
I20260926 05:51:15.651849 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.651715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53628)
0926 05:51:15.651775 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.651818 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:51:15.652305 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.652180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53642)
0926 05:51:15.652251 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.652281 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 05:51:15.652436 2439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.61:34927: Network error: Client connection negotiation failed: client connection to 127.11.247.61:34927: connect: Connection refused (error 111)
I20260926 05:51:15.656893 2446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:15.659106 2446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:15.659227 2446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:15.663364 2446 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.61:34927
I20260926 05:51:15.663398 2513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.61:34927 every 8 connection(s)
I20260926 05:51:15.663784 2446 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/data/info.pb
I20260926 05:51:15.664845 2514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:15.666566 2514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:15.668080 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 2446
I20260926 05:51:15.668190 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-1/wal/instance
I20260926 05:51:15.668409 12252 external_mini_cluster.cc:1428] Setting key 24b51dd78f5d28d11e9a855d6235e4f0
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.247.60@KRBTEST.COM; defaulting to no policy
I20260926 05:51:15.677065 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.667423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.11.247.62:42137)
0926 05:51:15.667474 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.667476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.667686 (+ 210us) server_negotiation.cc:407] Connection header received
0926 05:51:15.668188 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.668190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.668202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.668287 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.669056 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.669840 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.671358 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.671592 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.671680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.673960 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.673963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.673963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.673972 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.675546 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.676028 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.676029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.676030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.676113 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.676430 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.676432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.676432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.676583 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.676651 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.676920 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.676994 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:15.677358 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.667360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42137 (local address 127.0.0.1:34214)
0926 05:51:15.667556 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.667599 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.668080 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.668383 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.668387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.668711 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.668931 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.668935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.669993 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.669994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.671259 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.671262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.671726 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.671728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.671846 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.672427 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.672437 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.673840 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.675657 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.675666 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.675668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.675940 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.676191 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.676192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.676193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.676338 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.676656 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.676661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.676795 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.676885 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.677021 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":120,"threads_started":1}
I20260926 05:51:15.678046 2514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.247.60@KRBTEST.COM" created.
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:51:15.687705 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.678331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53644)
0926 05:51:15.678385 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.678397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.678459 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.678749 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.678750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.678931 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.679008 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.679011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.680009 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.680010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.680816 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.680819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.681548 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.681549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.681674 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.682854 (+ 1180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.682860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.684546 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.686508 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.686509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.686510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.686724 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.686963 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.686964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.686964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.687020 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.687346 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.687347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.687389 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.687457 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.687576 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:51:15.687774 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.678410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.11.247.61:34927)
0926 05:51:15.678562 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.678567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.678625 (+ 58us) server_negotiation.cc:407] Connection header received
0926 05:51:15.678651 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.678652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.678664 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.678720 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.679099 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.679910 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.680903 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.681468 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.681547 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.684658 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.684669 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.684670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.684677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.686401 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.686822 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.686825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.686826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.686889 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.687087 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.687088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.687089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.687265 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.687358 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.687507 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.687651 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260926 05:51:15.688529 2514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:15.688891 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.688789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49324)
0926 05:51:15.688828 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.688871 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 05:51:15.689034 2462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
I20260926 05:51:15.689497 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.689375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49338)
0926 05:51:15.689448 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.689474 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 05:51:15.689610 2514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34373: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.247.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab.
Entry for principal kudu/127.11.247.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab.
I20260926 05:51:15.698534 12252 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksryJ5D/build/release/bin/kudu
/tmp/dist-test-tasksryJ5D/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:34373
--webserver_interface=127.11.247.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab
--principal=kudu/127.11.247.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.247.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.247.62:42137,127.11.247.61:34927,127.11.247.60:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:51:15.738855 2514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } attempt: 1
I20260926 05:51:15.739375 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.739230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49344)
0926 05:51:15.739299 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.739343 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 05:51:15.739734 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.739635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49350)
0926 05:51:15.739685 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.739710 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:51:15.739832 2514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34373: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
I20260926 05:51:15.743563 2439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } attempt: 3
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:51:15.752573 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.744046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53658)
0926 05:51:15.744134 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.744149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.744218 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.744383 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.744384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.744540 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.744610 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.744612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.745623 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.745625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.746193 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.746195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.746778 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.746780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.746856 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.747439 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.747446 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.749380 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.751465 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.751467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.751467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.751658 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.751886 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.751887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.751887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.751967 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.752276 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.752277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.752330 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.752401 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.752495 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 05:51:15.752646 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.744165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.11.247.61:34927)
0926 05:51:15.744243 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.744246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.744258 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:51:15.744289 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.744290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.744304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.744369 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.744708 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.745482 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.746292 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.746681 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.746765 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.749490 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.749492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.749493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.749502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.751358 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.751750 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.751751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.751751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.751817 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.752031 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.752032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.752032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.752179 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.752247 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.752442 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.752529 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:51:15.753222 2439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:15.753649 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.753532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49354)
0926 05:51:15.753588 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.753617 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:51:15.754047 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.753961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49366)
0926 05:51:15.754006 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.754030 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:51:15.754163 2439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34373: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
I20260926 05:51:15.785373 2439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } attempt: 1
I20260926 05:51:15.785895 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.785733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49374)
0926 05:51:15.785804 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.785855 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:51:15.786372 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.786265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49386)
0926 05:51:15.786315 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.786348 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 05:51:15.786547 2439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34373: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
W20260926 05:51:15.787791 2521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:51:15.788033 2521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:51:15.788053 2521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:51:15.788128 2521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:51:15.789426 2521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:51:15.789471 2521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:51:15.789486 2521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:51:15.789500 2521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:51:15.789543 2521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:51:15.791359 2521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.247.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.247.62:42137,127.11.247.61:34927,127.11.247.60:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/krb5kdc/kudu_127.11.247.60.keytab
--principal=kudu/127.11.247.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.247.60:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.247.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:51:15.791658 2521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:51:15.791918 2521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:51:15.797351 2521 server_base.cc:1034] running on GCE node
W20260926 05:51:15.797264 2526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:15.797271 2527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:51:15.797520 2529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:51:15.797788 2521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:51:15.798152 2521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:51:15.799332 2521 hybrid_clock.cc:648] HybridClock initialized: now 1790401875799309 us; error 42 us; skew 500 ppm
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:51:15.802059 2521 init.cc:377] Logged in from keytab as kudu/127.11.247.60@KRBTEST.COM (short username kudu)
I20260926 05:51:15.802541 2521 webserver.cc:460] Webserver started at http://127.11.247.60:37947/ using document root <none> and password file <none>
I20260926 05:51:15.802807 2521 fs_manager.cc:359] Metadata directory not provided
I20260926 05:51:15.802886 2521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:51:15.803038 2521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:51:15.804203 2521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data/instance:
uuid: "d6f860d22a1d450bada739016158cfee"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "dd8777a4e99ed3f4eabd8d4b81865a05"
server_key_iv: "7e087fb23dcc40781fda7149a6b65bff"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.804605 2521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal/instance:
uuid: "d6f860d22a1d450bada739016158cfee"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "dd8777a4e99ed3f4eabd8d4b81865a05"
server_key_iv: "7e087fb23dcc40781fda7149a6b65bff"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.806070 2521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:51:15.807051 2536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:51:15.807301 2521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:51:15.807436 2521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data,/tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal
uuid: "d6f860d22a1d450bada739016158cfee"
format_stamp: "Formatted at 2026-09-26 05:51:15 on dist-test-slave-21p4"
server_key: "dd8777a4e99ed3f4eabd8d4b81865a05"
server_key_iv: "7e087fb23dcc40781fda7149a6b65bff"
server_key_version: "encryptionkey@0"
I20260926 05:51:15.807524 2521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:51:15.807981 2514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } attempt: 2
I20260926 05:51:15.808504 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.808330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49388)
0926 05:51:15.808398 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.808460 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:51:15.809046 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.808937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49400)
0926 05:51:15.808993 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.809023 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 05:51:15.809201 2514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.247.60:34373: Network error: Client connection negotiation failed: client connection to 127.11.247.60:34373: connect: Connection refused (error 111)
I20260926 05:51:15.824875 2521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:51:15.827394 2521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:51:15.827549 2521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:51:15.831805 2521 rpc_server.cc:307] RPC server started. Bound to: 127.11.247.60:34373
I20260926 05:51:15.831902 2588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.247.60:34373 every 8 connection(s)
I20260926 05:51:15.832245 2521 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/data/info.pb
I20260926 05:51:15.833395 2589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:51:15.834515 12252 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksryJ5D/build/release/bin/kudu as pid 2521
I20260926 05:51:15.834684 12252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksryJ5D/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401595249435-12252-0/minicluster-data/master-2/wal/instance
I20260926 05:51:15.834962 12252 external_mini_cluster.cc:1428] Setting key f7ad5d8ec3b4f9dec097a761abac702f
I20260926 05:51:15.835275 2589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:51:15.845302 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.836130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.11.247.62:42137)
0926 05:51:15.836184 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.836186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.836359 (+ 173us) server_negotiation.cc:407] Connection header received
0926 05:51:15.836503 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.836504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.836517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.836585 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.837196 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.837667 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.838729 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.839042 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.839107 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.841691 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.841692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.841693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.841700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.843534 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.844091 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.844093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.844093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.844157 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.844499 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.844500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.844500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.844663 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.844729 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.845116 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.845198 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:15.845546 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.836034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.62:42137 (local address 127.0.0.1:34216)
0926 05:51:15.836232 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.836274 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.836399 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.836635 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.836637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.836914 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.837077 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.837081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.837750 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.837751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.838648 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.838650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.839144 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.839145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.839275 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.839970 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.839978 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.841585 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.843624 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.843632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.843633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.844011 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.844279 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.844280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.844280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.844426 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.844766 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.844775 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.844964 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.845064 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.845264 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260926 05:51:15.845546 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.835567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.11.247.60:34373)
0926 05:51:15.835746 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.835755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.835787 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:51:15.836365 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.836374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.836407 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.836496 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.837164 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.838117 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.839028 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.839465 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.839531 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.842252 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.842275 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.842275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.842282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.844180 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.844580 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.844583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.844584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.844635 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.844915 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.844916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.844916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.845042 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.845211 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.845313 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.845389 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260926 05:51:15.846274 2589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.62@KRBTEST.COM
I20260926 05:51:15.856714 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.846619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.11.247.61:34927)
0926 05:51:15.846669 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.846671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.846678 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:15.846703 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.846704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.846714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.846770 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.847140 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.847895 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.849078 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.849491 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.849577 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.852710 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.852712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.852712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.852721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.855309 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.855807 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.855809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.855809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.855889 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.856116 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.856117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.856118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.856253 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.856333 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.856516 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.856604 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:15.856714 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.846537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.61:34927 (local address 127.0.0.1:53662)
0926 05:51:15.846574 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.846606 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.846639 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.846784 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.846785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.846953 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.846999 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.847000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.847984 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.847985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.848991 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.848994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.849575 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.849576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.849667 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.850127 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.850132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.852605 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.855441 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.855443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.855444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.855692 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.855977 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.855978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.855978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.856037 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.856336 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.856337 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.856388 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.856474 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.856604 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:51:15.856916 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.846508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.11.247.62:42137)
0926 05:51:15.846546 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.846548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.846558 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:51:15.846656 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.846657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.846667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.846718 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.847301 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.848015 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.849172 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.849559 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.849626 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.853312 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.853314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.853315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.853323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.855347 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.855834 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.855835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.855835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.855913 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.856159 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.856160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.856161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.856314 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.856476 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.856719 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.856818 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:51:15.858322 2589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:51:15.861789 2439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } attempt: 2
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.60@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.247.61@KRBTEST.COM
I20260926 05:51:15.867059 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.858653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49426)
0926 05:51:15.858689 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.858700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.858729 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.858847 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.858848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.858998 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.859034 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.859036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.859637 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.859638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.860370 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.860378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.860822 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.860823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.860868 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.861514 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.861519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.863231 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.865790 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.865792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.865792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.866028 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.866232 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.866233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.866233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.866299 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.866623 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.866625 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.866679 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.866751 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.866866 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 05:51:15.867113 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.858671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.11.247.60:34373)
0926 05:51:15.858708 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.858710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.858733 (+ 23us) server_negotiation.cc:407] Connection header received
0926 05:51:15.858778 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.858779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.858786 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.858835 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.859102 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.859556 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.860452 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.860751 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.860868 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.863322 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.863324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.863324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.863332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.865450 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.866096 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.866098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.866098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.866159 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.866376 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.866377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.866377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.866518 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.866637 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.866800 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.866932 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":19}
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.62@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:51:15.867583 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.857891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.11.247.61:34927)
0926 05:51:15.857931 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.857933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.857940 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:15.857960 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.857960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.857969 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.858015 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.858695 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.859352 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.860298 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.860611 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.860686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.863914 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.863916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.863916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.863925 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.865694 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.866148 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.866149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.866149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.866209 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.866423 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.866424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.866425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.866545 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.866776 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.866982 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.867062 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:15.869026 2589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: Bootstrap starting.
I20260926 05:51:15.869791 2589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:15.870117 2589 log.cc:824] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:15.870767 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.862171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49442)
0926 05:51:15.862281 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.862291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.862329 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.862647 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.862649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.862864 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.862928 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.862930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.863970 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.863971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.864499 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.864501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.865551 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.865553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.865635 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.866055 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.866060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.867562 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.869509 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.869510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.869510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.869751 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.870025 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.870026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.870026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.870096 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.870392 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.870393 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.870443 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.870531 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.870660 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 05:51:15.870857 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.862275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.11.247.60:34373)
0926 05:51:15.862455 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.862457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.862464 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:15.862485 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.862486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.862497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.862572 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.863304 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.863883 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.864587 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.865466 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.865532 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.867645 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.867777 (+ 132us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.867778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.867785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.869412 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.869873 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.869874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.869875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.869942 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.870169 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.870170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.870171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.870317 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.870400 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.870571 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.870703 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260926 05:51:15.871124 2589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: No bootstrap required, opened a new log
I20260926 05:51:15.872445 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480: Bootstrap starting.
I20260926 05:51:15.872607 2589 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.872758 2589 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f860d22a1d450bada739016158cfee, State: Initialized, Role: FOLLOWER
I20260926 05:51:15.872851 2589 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.873175 2439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:15.873152 2596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.873226 2589 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:15.873238 2596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.873497 2439 log.cc:824] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:15.874320 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480: No bootstrap required, opened a new log
I20260926 05:51:15.875495 12252 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 05:51:15.875522 2608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:15.875649 2608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:15.876348 2439 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.876533 2439 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0516847e6e24030931ef2cfd29ab480, State: Initialized, Role: FOLLOWER
I20260926 05:51:15.876626 2439 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.876922 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.877002 2439 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:15.877020 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.875536 2521 master_runner.cc:186] Error getting master registration for 127.11.247.62:42137: OK, instance_id { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" instance_seqno: 1790401875509670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:51:15.877960 2521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:51:15.876423 2413 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c0516847e6e24030931ef2cfd29ab480"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34226
I20260926 05:51:15.878358 2413 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 05:51:15.878552 2413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 0 FOLLOWER]: Advancing to term 1
W20260926 05:51:15.879007 2621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:51:15.879112 2621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:51:15.879503 2413 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.879798 2413 leader_election.cc:288] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [CANDIDATE]: Term 1 election: Requested vote from peers 13528de440eb41bb9c50425c7c894d73 (127.11.247.61:34927), d6f860d22a1d450bada739016158cfee (127.11.247.60:34373)
I20260926 05:51:15.880133 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0516847e6e24030931ef2cfd29ab480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d6f860d22a1d450bada739016158cfee"
I20260926 05:51:15.880257 2563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:15.881325 2563 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0516847e6e24030931ef2cfd29ab480 in term 1.
I20260926 05:51:15.881597 2387 leader_election.cc:302] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0516847e6e24030931ef2cfd29ab480, d6f860d22a1d450bada739016158cfee; no voters:
I20260926 05:51:15.881728 2610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:51:15.881896 2610 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [term 1 LEADER]: Becoming Leader. State: Replica: c0516847e6e24030931ef2cfd29ab480, State: Running, Role: LEADER
I20260926 05:51:15.881999 2610 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.882481 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0516847e6e24030931ef2cfd29ab480. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.882571 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:15.883905 2371 master_runner.cc:186] Error getting master registration for 127.11.247.61:34927: OK, instance_id { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" instance_seqno: 1790401875662092 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:51:15.886410 2371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:51:15.886618 2630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:51:15.887104 2630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:51:15.892366 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.881045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.11.247.61:34927)
0926 05:51:15.881092 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.881094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.881614 (+ 520us) server_negotiation.cc:407] Connection header received
0926 05:51:15.881975 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.881976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.881990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.882035 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.883592 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.884316 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.885904 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.886704 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.886757 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.889193 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.889195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.889195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.889204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.890962 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.891409 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.891410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.891411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.891473 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.891690 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.891691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.891692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.891851 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.891931 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.892166 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.892254 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:15.892516 2563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [term 1 FOLLOWER]: Refusing update from remote peer c0516847e6e24030931ef2cfd29ab480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:15.893128 2628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:15.895048 2596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0516847e6e24030931ef2cfd29ab480. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.895165 2596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.900332 2514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } attempt: 3
I20260926 05:51:15.901646 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.891282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.11.247.60:34373)
0926 05:51:15.891335 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.891337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.893534 (+ 2197us) server_negotiation.cc:407] Connection header received
0926 05:51:15.893626 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.893627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.893639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.893668 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.894235 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.895018 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.896178 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.896526 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.896571 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.898170 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.898171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.898172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.898179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.900178 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.900696 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.900698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.900698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.900758 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.900975 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.900976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.900976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.901135 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.901235 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.901473 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.901552 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 05:51:15 dist-test-slave-21p4 krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401875, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.247.61@KRBTEST.COM for kudu/127.11.247.60@KRBTEST.COM
I20260926 05:51:15.906226 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.906356 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:15.906458 2631 mvcc.cc:205] Tried to move back new op lower bound from 7333486083652489216 to 7333486083613605888. Current Snapshot: MvccSnapshot[applied={T|T < 7333486083652489216}]
I20260926 05:51:15.907127 2596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.907222 2596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.908385 2634 mvcc.cc:205] Tried to move back new op lower bound from 7333486083652489216 to 7333486083613605888. Current Snapshot: MvccSnapshot[applied={T|T < 7333486083652489216}]
I20260926 05:51:15.909794 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.900950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.11.247.60:34373)
0926 05:51:15.900992 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.900994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.901001 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:51:15.901018 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.901019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.901028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.901073 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.901470 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.902167 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.903133 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.903441 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.903505 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.905818 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.905819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.905820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.905827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.907687 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.908221 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.908223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.908223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.908293 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.908532 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.908533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.908533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.908677 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.908743 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.909606 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.909691 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:15.910209 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.900627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.247.60:34373 (local address 127.0.0.1:49456)
0926 05:51:15.900720 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 05:51:15.900732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:51:15.900852 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:51:15.901140 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:51:15.901142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:51:15.901327 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:51:15.901379 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.901381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.902232 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.902233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.903040 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:51:15.903045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.903560 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.903561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:51:15.903619 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.904096 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:51:15.904101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:51:15.905728 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:51:15.907898 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.907900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.907900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.908136 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.908363 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:51:15.908364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:51:15.908365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:51:15.908458 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:51:15.908774 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:51:15.908776 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:51:15.909134 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 05:51:15.909211 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:51:15.910093 (+ 882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 05:51:15.911027 2630 catalog_manager.cc:1347] Generated new cluster ID: 5d23015c73154bc7841fa4e6c6aede57
I20260926 05:51:15.911082 2630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:51:15.916173 2514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: Bootstrap starting.
I20260926 05:51:15.916965 2514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: Neither blocks nor log segments found. Creating new log.
I20260926 05:51:15.917271 2514 log.cc:824] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: Log is configured to *not* fsync() on all Append() calls
I20260926 05:51:15.918618 2514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: No bootstrap required, opened a new log
I20260926 05:51:15.920845 2514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.921003 2514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13528de440eb41bb9c50425c7c894d73, State: Initialized, Role: FOLLOWER
I20260926 05:51:15.921090 2514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } }
I20260926 05:51:15.921556 2487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:51:15.922248 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.890899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.11.247.62:42137)
0926 05:51:15.890953 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.890955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.891280 (+ 325us) server_negotiation.cc:407] Connection header received
0926 05:51:15.891450 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.891451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.891464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.891499 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.895611 (+ 4112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.896304 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.898174 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.898498 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.898544 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.908547 (+ 10003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:51:15.908549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:51:15.908549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:51:15.908558 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:51:15.910200 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.917567 (+ 7367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.917570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.917570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.917665 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:51:15.917958 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:51:15.917960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:51:15.917960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:51:15.918112 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:51:15.918221 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.922038 (+ 3817us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.922134 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:51:15.922566 2487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [term 1 FOLLOWER]: Refusing update from remote peer c0516847e6e24030931ef2cfd29ab480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:51:15.922645 2488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0516847e6e24030931ef2cfd29ab480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13528de440eb41bb9c50425c7c894d73"
I20260926 05:51:15.923159 2610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:51:15.923842 2488 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0516847e6e24030931ef2cfd29ab480 in term 1.
I20260926 05:51:15.921409 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.924131 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.926677 2630 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:51:15.927279 2630 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:51:15.921447 2514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:51:15.929152 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0516847e6e24030931ef2cfd29ab480. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.929569 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.934725 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.934814 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:51:15.935559 2610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0516847e6e24030931ef2cfd29ab480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0516847e6e24030931ef2cfd29ab480" member_type: VOTER last_known_addr { host: "127.11.247.62" port: 42137 } } peers { permanent_uuid: "13528de440eb41bb9c50425c7c894d73" member_type: VOTER last_known_addr { host: "127.11.247.61" port: 34927 } } peers { permanent_uuid: "d6f860d22a1d450bada739016158cfee" member_type: VOTER last_known_addr { host: "127.11.247.60" port: 34373 } } }
I20260926 05:51:15.935658 2610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480 [sys.catalog]: This master's current role is: LEADER
I20260926 05:51:15.942806 2630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0516847e6e24030931ef2cfd29ab480: Generated new TSK 0
I20260926 05:51:15.943053 2630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:51:15.943391 2654 catalog_manager.cc:1259] Loaded cluster ID: 5d23015c73154bc7841fa4e6c6aede57
I20260926 05:51:15.943456 2654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: loading cluster ID for follower catalog manager: success
I20260926 05:51:15.944686 2654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: acquiring CA information for follower catalog manager: success
I20260926 05:51:15.945078 2654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13528de440eb41bb9c50425c7c894d73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 05:51:15.959919 2518 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:15.958668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.11.247.61:34927)
0926 05:51:15.958720 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.958722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.959063 (+ 341us) server_negotiation.cc:407] Connection header received
0926 05:51:15.959186 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.959187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.959200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.959232 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.959798 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53698: BlockingRecv error: recv got EOF from 127.0.0.1:53698 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 05:51:15.961038 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:15.959201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.11.247.60:34373)
0926 05:51:15.959246 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.959248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.960320 (+ 1072us) server_negotiation.cc:407] Connection header received
0926 05:51:15.960496 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.960498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.960511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.960546 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.960967 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49472: BlockingRecv error: recv got EOF from 127.0.0.1:49472 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:51:15.961213 2443 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:15.958707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.11.247.62:42137)
0926 05:51:15.958750 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.958753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.959368 (+ 615us) server_negotiation.cc:407] Connection header received
0926 05:51:15.959467 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.959469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.959481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.959511 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.961141 (+ 1630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34254: BlockingRecv error: recv got EOF from 127.0.0.1:34254 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:51:15.970412 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.962821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.11.247.61:34927)
0926 05:51:15.962859 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.962861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.966833 (+ 3972us) server_negotiation.cc:407] Connection header received
0926 05:51:15.966965 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.966967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.967018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.967064 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:15.967315 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.968088 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.969323 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.969620 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.969663 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.969900 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:15.970081 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:15.970146 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.970296 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.970321 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 05:51:15.974170 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:15.973234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.11.247.60:34373)
0926 05:51:15.973287 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.973289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.973514 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:51:15.973653 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.973654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.973673 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.973703 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.974108 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49478: BlockingRecv error: recv got EOF from 127.0.0.1:49478 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:51:15.980232 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.976533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.11.247.61:34927)
0926 05:51:15.976594 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.976596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.976927 (+ 331us) server_negotiation.cc:407] Connection header received
0926 05:51:15.977025 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.977026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.977037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.977082 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:15.977336 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.978127 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.979440 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.979729 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.979770 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.979961 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:15.979998 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:15.980024 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.980125 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.980147 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:51:15.983649 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:15.982837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.11.247.60:34373)
0926 05:51:15.982887 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.982889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.983068 (+ 179us) server_negotiation.cc:407] Connection header received
0926 05:51:15.983171 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.983172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.983184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.983214 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.983586 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49480: BlockingRecv error: recv got EOF from 127.0.0.1:49480 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:51:15.990399 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.986985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.11.247.61:34927)
0926 05:51:15.987034 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.987036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.987330 (+ 294us) server_negotiation.cc:407] Connection header received
0926 05:51:15.987435 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.987436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.987448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.987478 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:15.987677 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.988455 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.989591 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:15.989890 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:15.989931 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:15.990118 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:15.990153 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:15.990179 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:15.990292 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:51:15.990312 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:51:15.993887 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:15.992954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.11.247.60:34373)
0926 05:51:15.992997 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.992999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.993269 (+ 270us) server_negotiation.cc:407] Connection header received
0926 05:51:15.993378 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.993379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.993392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.993422 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:15.993826 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49488: BlockingRecv error: recv got EOF from 127.0.0.1:49488 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:51:16.002934 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:15.999480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.11.247.61:34927)
0926 05:51:15.999527 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:15.999529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:15.999809 (+ 280us) server_negotiation.cc:407] Connection header received
0926 05:51:15.999923 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:15.999924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:15.999936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:15.999965 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.000169 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.000895 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.002018 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.002319 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.002368 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.002614 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.002654 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.002684 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.002822 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.002846 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:51:16.006563 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.005731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.11.247.60:34373)
0926 05:51:16.005787 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.005789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.005930 (+ 141us) server_negotiation.cc:407] Connection header received
0926 05:51:16.006035 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.006036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.006050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.006080 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.006497 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49496: BlockingRecv error: recv got EOF from 127.0.0.1:49496 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:51:16.019500 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:16.016054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.11.247.61:34927)
0926 05:51:16.016101 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.016103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.016399 (+ 296us) server_negotiation.cc:407] Connection header received
0926 05:51:16.016508 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.016509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.016520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.016550 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.016748 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.017494 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.018657 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.018945 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.018993 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.019205 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.019244 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.019285 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.019396 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.019418 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:51:16.025972 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.024746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.11.247.60:34373)
0926 05:51:16.024793 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.024796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.025084 (+ 288us) server_negotiation.cc:407] Connection header received
0926 05:51:16.025196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.025197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.025210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.025241 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.025654 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49498: BlockingRecv error: recv got EOF from 127.0.0.1:49498 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:16.054411 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:16.050962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.11.247.61:34927)
0926 05:51:16.051011 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.051013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.051321 (+ 308us) server_negotiation.cc:407] Connection header received
0926 05:51:16.051400 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.051402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.051414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.051488 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.051624 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.052366 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.053451 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.053882 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.053965 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.054123 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.054162 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.054219 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.054303 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.054326 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:51:16.057348 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.056569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.11.247.60:34373)
0926 05:51:16.056615 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.056617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.056748 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:51:16.056870 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.056871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.056884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.056915 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.057283 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49508: BlockingRecv error: recv got EOF from 127.0.0.1:49508 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:51:16.094544 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:16.091127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.11.247.61:34927)
0926 05:51:16.091177 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.091180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.091472 (+ 292us) server_negotiation.cc:407] Connection header received
0926 05:51:16.091587 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.091588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.091600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.091630 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.091829 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.092587 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.093736 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.093988 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.094033 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.094241 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.094279 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.094319 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.094428 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.094451 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:51:16.097453 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.096703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.11.247.60:34373)
0926 05:51:16.096771 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.096773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.096840 (+ 67us) server_negotiation.cc:407] Connection header received
0926 05:51:16.096943 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.096944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.096957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.096987 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.097387 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49514: BlockingRecv error: recv got EOF from 127.0.0.1:49514 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:51:16.165644 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:16.163025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.11.247.61:34927)
0926 05:51:16.163110 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.163112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.163243 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:51:16.163307 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.163308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.163319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.163372 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.163506 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.164021 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.164919 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.165115 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.165173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.165343 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.165373 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.165417 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.165527 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.165544 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 05:51:16.168313 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.167633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.11.247.60:34373)
0926 05:51:16.167682 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.167684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.167730 (+ 46us) server_negotiation.cc:407] Connection header received
0926 05:51:16.167798 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.167799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.167828 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.167871 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.168209 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49518: BlockingRecv error: recv got EOF from 127.0.0.1:49518 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:51:16.300276 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:16.297494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.11.247.61:34927)
0926 05:51:16.297593 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.297595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.297610 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:51:16.297694 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.297695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.297704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.297771 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.297993 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.298485 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.299439 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.299643 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.299709 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.299918 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.299950 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.299999 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.300176 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.300198 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 05:51:16.302837 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.302116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.11.247.60:34373)
0926 05:51:16.302198 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.302200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.302253 (+ 53us) server_negotiation.cc:407] Connection header received
0926 05:51:16.302319 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.302321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.302333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.302386 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.302767 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49526: BlockingRecv error: recv got EOF from 127.0.0.1:49526 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:51:16.562629 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:16.560156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.11.247.61:34927)
0926 05:51:16.560246 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.560249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.560257 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:16.560288 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.560289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.560304 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.560373 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:16.560544 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.561042 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.561839 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:16.562079 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:16.562164 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:16.562326 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:16.562363 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:16.562423 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:16.562487 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:51:16.562507 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 05:51:16.565145 2592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:51:16.564548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.11.247.60:34373)
0926 05:51:16.564648 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:16.564650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:16.564658 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:51:16.564690 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:16.564691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:16.564701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:16.564773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:51:16.565089 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49536: BlockingRecv error: recv got EOF from 127.0.0.1:49536 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 05:51:16.876154 2608 catalog_manager.cc:1259] Loaded cluster ID: 5d23015c73154bc7841fa4e6c6aede57
I20260926 05:51:16.876250 2608 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: loading cluster ID for follower catalog manager: success
I20260926 05:51:16.877158 2608 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: acquiring CA information for follower catalog manager: success
I20260926 05:51:16.877523 2608 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6f860d22a1d450bada739016158cfee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:51:17.081261 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:17.078482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.11.247.61:34927)
0926 05:51:17.078662 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:17.078665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:17.078676 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:51:17.078785 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:17.078786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:17.078799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:17.078873 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:17.079006 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.079599 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.080381 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.080628 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.080696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:17.080909 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:17.080946 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:17.081006 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:17.081148 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 05:51:17.081174 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260926 05:51:17.085783 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:17.083110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.11.247.60:34373)
0926 05:51:17.083238 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:17.083240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:17.083249 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:17.083289 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:17.083290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:17.083370 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:17.083459 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:17.083575 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.084094 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.084880 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.085115 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.085196 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:17.085377 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:17.085508 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:17.085603 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:17.085642 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:51:17.085667 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":48,"threads_started":1}
I20260926 05:51:17.090898 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:17.087704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.11.247.61:34927)
0926 05:51:17.087775 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:17.087778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:17.088079 (+ 301us) server_negotiation.cc:407] Connection header received
0926 05:51:17.088144 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:17.088145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:17.088157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:17.088203 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:17.088402 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.089250 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.090033 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.090330 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.090368 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:17.090570 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:17.090635 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:17.090683 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:17.090781 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:51:17.090812 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:51:17.091106 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:17.087677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.11.247.62:42137)
0926 05:51:17.087830 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:17.087832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:17.087841 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:51:17.087886 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:17.087893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:17.087945 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:17.088051 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:17.088183 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.088856 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.089903 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.090202 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.090263 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:17.090518 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:17.090692 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:17.090777 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:17.090958 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 05:51:17.090991 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260926 05:51:17.091399 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:51:17.087850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.11.247.60:34373)
0926 05:51:17.087891 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:51:17.087893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:51:17.088215 (+ 322us) server_negotiation.cc:407] Connection header received
0926 05:51:17.088245 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:51:17.088246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:51:17.088255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:51:17.088335 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:51:17.088468 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.089200 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.090264 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:51:17.090552 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:51:17.090658 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:51:17.090833 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:51:17.090866 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:51:17.090931 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:51:17.091284 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 05:51:17.091311 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:51:17.093122 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 2371
I20260926 05:51:17.098351 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 2446
I20260926 05:51:17.103696 12252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksryJ5D/build/release/bin/kudu with pid 2521
2026-09-26T05:51:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2908 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5929 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281862 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST